Medical image processing device, medical image processing method and program

Abstract

To provide a medical image processing device that can generate images more suitable for image diagnosis than conventional ones.SOLUTION: A medical image processing device comprises: an acquisition unit for acquiring a first image which is a medical image of a prescribed portion of a subject; a high-quality image enhancement unit for generating, from the first image, a second image whose image quality is enhanced compared to the first image, using a high-quality image enhancement engine including a machine learning engine; and a display control unit for causing a display unit to display a composite image obtained by combining the first image and the second image based on a ratio obtained by using information regarding at least some regions in the first image.SELECTED DRAWING: Figure 34

Description

[Technical Field]

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art In the medical field, images are acquired by various imaging devices and used for diagnostic imaging by medical professionals in order to identify diseases in subjects and observe the severity of the diseases. Types of imaging devices include, for example, in the field of radiology, X-ray imaging devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices, etc. Also, in the field of ophthalmology, for example, types of imaging devices include fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

[0003] To perform accurate diagnostic imaging and complete the procedure quickly, it is important that the images acquired by the imaging device have high image quality, such as low noise, high resolution and spatial resolution, and appropriate gradation. Images that highlight the area or lesion you are interested in can also be useful.

[0004] However, in many imaging devices, some trade-off is required to obtain images suitable for diagnostic imaging, such as high image quality. For example, one method for obtaining high-quality images is to purchase a high-performance imaging device, but this often requires a larger investment than a low-performance device.

[0005] Furthermore, for example, in CT, the patient's radiation exposure may need to be increased to obtain images with less noise. For example, in MRI, contrast agents with potential side effects may be used to obtain images that highlight the area of ​​interest. For example, in OCT, the imaging time may be longer if the imaging area is large or high spatial resolution is required. Furthermore, some imaging devices require multiple imaging sessions to obtain high-quality images, which increases the imaging time.

[0006] Patent Document 1 discloses a technology that uses an artificial intelligence engine to convert previously acquired images into images with higher resolution in order to respond to rapid advances in medical technology and the need for simple imaging in emergencies. With this technology, for example, it is possible to convert images acquired through simple imaging with little cost into images with higher resolution. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2018-5841 Summary of the Invention [Problem to be solved by the invention]

[0008] However, even high-resolution images may not be suitable for diagnostic imaging. For example, even high-resolution images may not be able to accurately grasp the object to be observed if they have a lot of noise or low contrast.

[0009] In response to this, one of the objects of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program that can generate images that are more suitable for image diagnosis than conventional methods. [Means for solving the problem]

[0010] A medical image processing device according to one embodiment of the present invention comprises an acquisition unit that acquires a first image, which is a medical image of a specific part of a subject; an image quality improvement unit that uses an image quality improvement engine including a machine learning engine to generate a second image from the first image, the second image having higher image quality than the first image; and a display control unit that displays a composite image obtained by combining the first image and the second image using a ratio obtained using information regarding at least a portion of the first image on a display unit.

[0011] In addition, a medical image processing method according to another embodiment of the present invention includes acquiring a first image, which is a medical image of a specific part of a subject; using an image quality improvement engine including a machine learning engine to generate a second image from the first image, which has higher image quality than the first image; and displaying a composite image obtained by combining the first image and the second image using a ratio obtained using information regarding at least a portion of the first image on a display unit. [Effects of the Invention]

[0012] According to one aspect of the present invention, it is possible to generate an image that is more suitable for image diagnosis than conventional methods. [Brief explanation of the drawings]

[0013] [Figure 1] 1 shows an example of the configuration of a neural network related to image quality improvement processing. [Figure 2] 10 shows an example of the configuration of a neural network related to the shooting location estimation process. [Figure 3] 1 shows an example of the configuration of a neural network related to image authenticity evaluation processing. [Figure 4] 1 shows an example of a schematic configuration of an image processing device according to a first embodiment. [Figure 5] FIG. 3 is a flowchart showing an example of the flow of image processing according to the first embodiment. [Figure 6] FIG. 10 is a flowchart showing another example of the flow of image processing according to the first embodiment. [Figure 7]FIG. 10 is a flowchart showing an example of the flow of image processing according to the second embodiment. [Figure 8] FIG. 10 is a diagram for explaining image processing according to a fourth embodiment. [Figure 9] FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fourth embodiment. [Figure 10] FIG. 13 is a diagram for explaining image processing according to the fifth embodiment. [Figure 11] FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fifth embodiment. [Figure 12] FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. [Figure 13] FIG. 20 is a flowchart showing an example of the flow of image quality improvement processing according to the sixth embodiment. [Figure 14] FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. [Figure 15] 13 shows an example of a schematic configuration of an image processing device according to a seventh embodiment. [Figure 16] FIG. 13 is a flowchart showing an example of the flow of image processing according to the seventh embodiment. [Figure 17] 13 shows an example of a user interface according to the seventh embodiment. [Figure 18] 13 shows an example of a schematic configuration of an image processing device according to a ninth embodiment. [Figure 19] FIG. 13 is a flowchart showing an example of the flow of image processing according to the ninth embodiment. [Figure 20] 20 shows an example of a schematic configuration of an image processing device according to a twelfth embodiment. [Figure 21A] FIG. 22 is a flowchart showing an example of the flow of image quality improvement processing according to the thirteenth embodiment. [Figure 21B] FIG. 22 is a flowchart showing another example of the flow of the image quality improvement processing according to the thirteenth embodiment. [Figure 22] 23 shows an example of a schematic configuration of an image processing device according to a seventeenth embodiment. [Figure 23] FIG. 22 is a flowchart showing an example of the flow of image processing according to the seventeenth embodiment. [Figure 24] 1 shows an example of the configuration of a neural network related to image quality improvement processing. [Figure 25] 23 shows an example of a schematic configuration of an image processing device according to a nineteenth embodiment. [Figure 26] FIG. 22 is a flowchart showing an example of the flow of image processing according to the nineteenth embodiment. [Figure 27] FIG. 21 is a flowchart showing an example of the flow of image processing according to the twenty-first embodiment. [Figure 28] 10 shows an example of a teacher image related to image quality improvement processing. [Figure 29] 1 shows an example of an input image for image quality improvement processing. [Figure 30] 22 shows an example of a schematic configuration of an image processing device according to a 22nd embodiment. [Figure 31] FIG. 22 is a flowchart showing an example of the flow of image processing according to the twenty-second embodiment. [Figure 32] FIG. 22 is a diagram for explaining a wide-angle image according to the twenty-second embodiment. [Figure 33] FIG. 23 is a diagram for explaining image quality improvement processing according to the 23rd embodiment. [Figure 34] 24 shows an example of a user interface according to the twenty-fourth embodiment. [Figure 35] 25 shows an example of a schematic configuration of an image processing device according to a 25th embodiment. [Figure 36] 13 shows an example of the configuration of a neural network used as a machine learning engine according to Modification 6. [Figure 37] 13 shows an example of the configuration of a neural network used as a machine learning engine according to Modification 6. [Figure 38] 24 shows an example of a user interface according to the twenty-fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Exemplary embodiments for carrying out the present invention will now be However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed depending on the configuration of the device to which the present invention is applied or various conditions. In addition, the same reference numerals are used in the drawings to indicate identical or functionally similar elements.

[0015] <Terminology> First, the terms used in this specification will be explained.

[0016] In the network herein, the devices may be connected by wired or wireless lines. Here, the lines connecting the devices in the network include, for example, dedicated lines, local area network (hereinafter referred to as LAN) lines, wireless LAN lines, internet lines, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like.

[0017] A medical image processing apparatus may be configured by two or more devices capable of communicating with each other, or may be configured by a single device. Furthermore, each component of the medical image processing apparatus may be configured by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Furthermore, each component may be configured by a circuit that performs a specific function, such as an ASIC. Furthermore, it may be configured by a combination of any other hardware and any software.

[0018] Furthermore, medical images processed by the medical image processing apparatus or medical image processing method according to the following embodiments include images acquired using any modality (imaging apparatus, imaging method). The medical images to be processed may include medical images acquired by any imaging device or images created by a medical image processing device or medical image processing method according to the following embodiments.

[0019] Furthermore, the medical image to be processed is an image of a predetermined part of the subject (examinee), and the image of the predetermined part includes at least a portion of the predetermined part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black-and-white image or a color image. The medical image may also be an image representing the structure (morphology) of the predetermined part or an image representing its function. Images representing function include, for example, images representing hemodynamics (blood flow rate, blood flow velocity, etc.), such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. The predetermined part of the subject may be determined according to the subject to be imaged, and may include any part, such as the human eye (examined eye), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, as well as the head, chest, legs, and arms.

[0020] Furthermore, the medical image may be a tomographic image of the subject, or a front image. Front images include, for example, a front image of the fundus, a front image of the anterior segment, a fundus image photographed by fluorescence, and an En-Face image generated using data acquired by OCT (three-dimensional OCT data) covering at least a partial range in the depth direction of the subject. The En-Face image may also be an OCTA En-Face image (motion contrast front image) generated using data from three-dimensional OCTA data (three-dimensional motion contrast data) covering at least a partial range in the depth direction of the subject. Three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0021] Furthermore, an imaging device is a device for capturing an image used for diagnosis. Imaging devices include, for example, devices that obtain an image of a predetermined part of a subject by irradiating the predetermined part with light, radiation such as X-rays, electromagnetic waves, ultrasound, or the like, and devices that obtain an image of the predetermined part by detecting radiation emitted from a subject. More specifically, imaging devices according to the following embodiments include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, an endoscope, and the like.

[0022] The OCT device may include a time-domain OCT (TD-OCT) device or a Fourier-domain OCT (FD-OCT) device. The Fourier-domain OCT device may include a spectral-domain OCT (SD-OCT) device or a swept-spectrum OCT (SS-OCT) device. The SLO device or OCT device may include a wavefront-compensated SLO (AO-SLO) device or a wavefront-compensated OCT (AO-OCT) device using wavefront adaptive optics. The SLO device or OCT device may also include a polarization-symmetric SLO (PS-SLO) device or a polarization-symmetric OCT (PS-OCT) device for visualizing information related to polarization phase difference and depolarization.

[0023] An image management system is a device or system that receives and stores images captured by an imaging device or processed images. The image management system can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS). In particular, the image management system according to the following embodiment includes a database that can store various information, such as subject information and imaging time, associated with received images. The image management system is connected to a network and can transmit and receive images, convert images, and transmit and receive various information associated with stored images in response to a request from other devices.

[0024] The imaging conditions are various pieces of information at the time of capturing an image acquired by an imaging device. The imaging conditions include, for example, information about the imaging device, information about the facility where the imaging was performed, information about the examination related to the imaging, information about the photographer, and information about the subject. The imaging conditions also include, for example, the imaging date and time, the name of the imaging region, the imaging area, the imaging angle of view, the imaging method, the image resolution and gradation, the image size, the applied image filter, information about the image data format, and information about the radiation dose. The imaging area may include a peripheral area shifted from a specific imaging region, an area including multiple imaging regions, etc.

[0025] The photographing conditions can be stored in a data structure constituting the image, as photographing condition data separate from the image, or in a database or image management system associated with the photographing device. Therefore, the photographing conditions can be acquired by a procedure corresponding to the photographing condition storage means of the photographing device. Specifically, the photographing conditions can be acquired, for example, by analyzing the data structure of the image output by the photographing device, acquiring photographing condition data corresponding to the image, or accessing an interface for acquiring photographing conditions from a database associated with the photographing device.

[0026] Depending on the imaging device, there may be some imaging conditions that cannot be acquired because they are not saved. For example, this may be the case if the imaging device does not have a function for acquiring or saving specific imaging conditions, or if such a function is disabled. Furthermore, there may be cases where the imaging conditions are not saved because they are considered to be unrelated to the imaging device or imaging. Furthermore, there may be cases where the imaging conditions are hidden, encrypted, or cannot be acquired without the necessary rights. However, there are cases where even unsaved imaging conditions can be acquired. For example, the name of the imaging part or imaging area can be identified by performing image analysis.

[0027] A machine learning model is a model that has been trained (learned) in advance using appropriate training data (learning data) for an arbitrary machine learning algorithm. The training data consists of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data in the pairs that make up the training data may be suitable for the desired configuration, such as one being an image and the other being a numerical value, one being composed of a group of images and the other being a character string, or both being images.

[0028] Specifically, for example, there is training data (hereinafter referred to as first training data) configured by a group of pairs of images acquired by OCT and imaging region labels corresponding to the images. The imaging region labels are unique numerical values ​​or character strings that represent the region. Another example of training data is training data (hereinafter referred to as second training data) configured by a group of pairs of noisy low-quality images acquired by normal OCT imaging and high-quality images acquired by multiple imaging by OCT and processed to improve image quality.

[0029] When input data is input to a machine learning model, output data is output according to the design of the machine learning model. The machine learning model outputs output data that is likely to correspond to the input data, for example, according to trends trained using training data. Furthermore, the machine learning model can, for example, output a numerical value representing the likelihood that each type of output data corresponds to the input data, for example, according to trends trained using training data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with first training data, the machine learning model outputs an imaging site label for the imaging site captured in the image, or outputs a probability for each imaging site label. Furthermore, for example, when a noisy, low-quality image acquired by normal OCT imaging is input to a machine learning model trained with second training data, the machine learning model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed to improve image quality. Note that, from the perspective of maintaining quality, the machine learning model can be configured not to use its own output data as training data.

[0030] Furthermore, machine learning algorithms include deep learning techniques such as convolutional neural networks (CNNs). In deep learning techniques, different parameter settings for layers and nodes constituting a neural network may affect the degree to which trends trained using training data can be reproduced in output data. For example, in a deep learning machine learning model using first training data, setting more appropriate parameters may increase the probability of outputting a correct imaging region label. Furthermore, in a deep learning machine learning model using second training data, setting more appropriate parameters may enable output of higher quality images.

[0031] Specifically, parameters in CNN can include, for example, the filter kernel size, number of filters, stride value, and dilation value set for the convolution layer, as well as the number of nodes output by the fully connected layer. The parameter set and the number of training epochs can be set to values ​​preferred for the usage of the machine learning model based on training data. For example, the parameter set and the number of epochs can be set based on training data to output the correct imaging region label with a higher probability or to output higher quality images.

[0032] Here is an example of a method for determining such parameter sets and the number of epochs. First, 70% of the pairs constituting the training data are randomly set as those for training, and the remaining 30% are set as those for evaluation. Next, the training pairs are used to train the machine learning model, and at the end of each training epoch, a training evaluation value is calculated using the evaluation pairs. The training evaluation value is, for example, the average value of a group of values ​​obtained by evaluating, using a loss function, the output when input data constituting each pair is input to the machine learning model being trained, and the output data corresponding to the input data. Finally, the parameter set and the number of epochs that result in the smallest training evaluation value are determined as the parameter set and the number of epochs for the machine learning model. Note that by dividing the pairs constituting the training data into those for training and those for evaluation and determining the number of epochs in this way, it is possible to prevent the machine learning model from overlearning the training pairs.

[0033] An image quality improvement engine (a trained model for image quality improvement) is a module that improves the image quality of an input low-quality image and outputs a high-quality image. In this specification, "improving image quality" refers to converting an input image into an image with image quality more suitable for diagnostic imaging, and a "high-quality image" refers to an image converted into an image with image quality more suitable for diagnostic imaging. Low-quality images include, for example, two-dimensional and three-dimensional images acquired by X-ray imaging, CT, MRI, OCT, PET, or SPECT, or three-dimensional moving images acquired by continuous CT imaging without settings specifically designed to achieve high image quality. Specifically, low-quality images include, for example, images acquired by low-dose imaging using an X-ray imaging device or CT, imaging using MRI without using a contrast agent, images acquired by short-time OCT imaging, and OCTA images acquired with a small number of imaging sessions.

[0034] Furthermore, the image quality suitable for diagnostic imaging depends on what is being diagnosed with each type of diagnostic imaging. Therefore, it is difficult to generalize, but image quality suitable for diagnostic imaging includes, for example, low noise, high contrast, displaying the subject in colors and gradations that are easy to observe, large image size, and high resolution. It also includes image quality in which non-existent objects and gradations that are drawn during the image generation process are removed from the image.

[0035] Furthermore, when high-quality images with low noise and high contrast are used for image analysis such as vascular analysis processing of images such as OCTA, or region segmentation processing of images such as CT or OCT, analysis can often be performed more accurately than when low-quality images are used. Therefore, high-quality images output by the image quality enhancement engine can be useful not only for image diagnosis but also for image analysis.

[0036] The image processing method constituting the image quality improvement method in the following embodiment performs processing using various machine learning algorithms such as deep learning. Note that the image processing method may perform any existing processing such as various image filter processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing in addition to processing using the machine learning algorithm.

[0037] In particular, an example of a CNN configuration for improving the image quality of two-dimensional images is shown in FIG. 1. This CNN configuration includes a group of multiple convolution processing blocks 100. The convolution processing blocks 100 include a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a rectifier linear unit (Rectifier Linear Unit). This CNN configuration also includes a merger layer 104 and a final convolution layer 105. The merger layer 104 combines output values ​​of the convolution processing blocks 100 with pixel values ​​constituting the image by concatenating or adding them together. The final convolution layer 105 outputs pixel values ​​constituting a high-image-quality image Im120, which are combined in the merger layer 104. In this configuration, the pixel values ​​constituting the input image Im110 are output through the convolution processing blocks 100 and then combined with the pixel values ​​constituting the input image Im110 in the combination layer 104. The combined pixel values ​​are then converted into a high-quality image Im120 in the final convolution layer 105.

[0038] For example, a certain level of image quality improvement can be achieved by setting the number of convolution processing blocks 100 to 16, setting the filter kernel size to 3 pixels wide and 3 pixels high, and setting the number of filters to 64 as parameters for the convolution layer 101 group. However, as described above in the description of the machine learning model, in practice, a better parameter group can be set using training data according to the usage form of the machine learning model. When three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

[0039] Note that when using some image processing methods, such as image processing using CNN, attention must be paid to image size. Specifically, to address issues such as the problem of high-resolution images not being sufficiently high-resolution at the periphery, it should be noted that different image sizes may be required for the input low-resolution image and the output high-resolution image.

[0040] For clarity, this will not be specified in the following embodiments. However, when a high-quality image engine is adopted that requires different image sizes for the image input to the high-quality image engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted by padding input images, such as images used as training data for training a machine learning model or images input to the high-quality image engine, or by combining the surrounding captured area of ​​the input image. Note that the padded area is filled with a fixed pixel value, filled with neighboring pixel values, or mirror padded, depending on the characteristics of the high-quality image enhancement method, so as to effectively improve image quality.

[0041] Furthermore, the image quality improvement method may be implemented using only one image processing method, or may be implemented using a combination of two or more image processing methods. Alternatively, multiple image quality improvement methods may be implemented in parallel to generate multiple high-quality image groups, and the highest quality image may be selected as the final high-quality image. The selection of the highest quality image may be performed automatically using an image quality assessment index, or may be performed in response to instructions from the examiner (user) by displaying multiple high-quality image groups on a user interface provided on an arbitrary display unit, etc.

[0042] In addition, since an input image that has not been subjected to image quality enhancement may be more suitable for image diagnosis, the input image may be included in the final image selection. Furthermore, parameters may be input to the image enhancement engine along with the low-quality image. For example, parameters specifying the degree of image quality enhancement or parameters specifying the image filter size used in the image processing method may be input to the image enhancement engine along with the input image.

[0043] The imaging location estimation engine is a module that estimates the imaging site and imaging area of ​​an input image. The imaging location estimation engine can output the location of the imaging site and imaging area depicted in the input image, or the probability that the imaging site and imaging area are the imaging site and imaging area for each imaging site label and imaging area label at the required level of detail.

[0044] Depending on the imaging device, the imaging site and imaging area may not be saved as imaging conditions, or may not be acquired and saved by the imaging device. Even if the imaging site and imaging area are saved, the required level of detail may not be saved. For example, the imaging site may be saved as "posterior segment," but it is unclear whether it is "macula," "optic disc," "macula and optic disc," or "other." In another example, the imaging site may be saved as "breast," but it is unclear whether it is "right breast," "left breast," or "both." Therefore, by using an imaging location estimation engine, it is possible to estimate the imaging site and imaging area of ​​the input image in these cases.

[0045] The image and data processing method constituting the estimation method of the imaging location estimation engine performs processing using various machine learning algorithms such as deep learning. Note that the image and data processing method may perform any existing estimation processing, such as natural language processing, matching processing using a database of similar images and data, knowledge-based processing, etc., in addition to or instead of processing using a machine learning algorithm. Note that the training data used to train the machine learning model constructed using the machine learning algorithm may be images labeled with the imaging site or imaging area. In this case, the images of the training data are used as input data, and the labels of the imaging site or imaging area are used as output data.

[0046] In particular, an example of a CNN configuration for estimating the shooting location of a two-dimensional image is the configuration shown in Figure 2. This CNN configuration includes a group of multiple convolution processing blocks 200, each of which is made up of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a normalized linear function. The CNN configuration also includes a final convolutional layer 204, a fully connected layer 205, and an output layer 206. The fully connected layer 205 fully connects the output value groups of the convolution processing block 200. The output layer 206 uses a Softmax function to output the probability of each expected imaging region label for the input image Im210 as an estimation result (Result) 207. With this configuration, for example, if the input image Im210 is an image of the "macular region," the highest probability is output for the imaging region label corresponding to the "macular region."

[0047] For example, the imaging region can be estimated with a certain degree of accuracy by setting the number of convolution processing blocks 200 to 16, the parameters of the convolution layer 201 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, as described above in the description of the machine learning model, in practice, a better parameter group can be set using training data according to the usage form of the machine learning model. When three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions. The estimation method may be implemented using only one image and data processing method, or may be implemented using a combination of two or more image and data processing methods.

[0048] The image quality assessment engine is a module that outputs an image quality assessment index for an input image. The image quality assessment processing method that calculates the image quality assessment index performs processing using various machine learning algorithms such as deep learning. Note that the image quality assessment processing method may also perform any existing evaluation processing, such as an image noise measurement algorithm and a matching process using a database of image quality assessment indices corresponding to similar images and base images. Note that these evaluation processes may be performed in addition to or instead of processing using a machine learning algorithm.

[0049] For example, the image quality assessment index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, pairs of input data constituting the training data for the machine learning model are image groups consisting of a group of low-quality images and a group of high-quality images captured in advance under various shooting conditions. Furthermore, pairs of output data constituting the training data for the machine learning model are, for example, image quality assessment index groups set for each of the image groups of the input data by an examiner performing image diagnosis.

[0050] The authenticity evaluation engine in the description of the present invention is a module that evaluates the rendering of an input image and evaluates with a certain degree of accuracy whether the image was captured and acquired by the target imaging device. The authenticity evaluation processing method performs processing using various machine learning algorithms such as deep learning. Note that the authenticity evaluation processing method may perform any existing evaluation processing such as knowledge-based processing in addition to or instead of processing using a machine learning algorithm.

[0051] For example, the authenticity evaluation process can be performed by a machine learning model constructed using a machine learning algorithm. First, the training data of the machine learning model will be described. The training data includes a group of pairs of high-quality images captured in advance under various shooting conditions and labels indicating that the images were captured and acquired using the target imaging device (hereinafter, "genuine labels"). The training data also includes a group of pairs of high-quality images generated by inputting low-quality images into an image quality improvement engine (first-level image quality improvement engine) and labels indicating that the images were not captured and acquired using the target imaging device (hereinafter, "fake labels"). A machine learning model trained using such training data outputs a fake label when a high-quality image generated by the first-level image quality improvement engine is input.

[0052] In particular, an example of the configuration of a CNN that performs authenticity assessment processing for two-dimensional images is shown in Figure 3. This CNN configuration includes a group of multiple convolution processing blocks 300, each of which is composed of a convolution layer 301, a batch normalization layer 302, and an activation layer 303 that uses a normalized linear function. This CNN configuration also includes a final convolution layer 304, a fully connected layer 305, and an output layer 306. The fully connected layer 305 fully connects the output values ​​of the convolution processing blocks 300. The output layer 306 also uses a sigmoid function to output a value of 1 (true) representing an authentic label or a value of 0 (false) representing a counterfeit label for the input image Im 310 as the result of the authenticity assessment processing (Result) 307.

[0053] Note that accurate authenticity assessment results can be obtained with a certain degree of precision by setting the number of convolution processing blocks 300 to 16, the parameters of the convolution layer 301 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, as described above in the description of the machine learning model, in practice, a better set of parameters can be set using training data according to the usage form of the machine learning model. Note that if three-dimensional or four-dimensional images need to be processed, the filter kernel size may be expanded to three or four dimensions.

[0054] The authenticity assessment engine may output an authentic label when it receives a high-resolution image generated by a second-level image enhancement engine that provides higher image quality than the first-level image enhancement engine. In other words, the authenticity assessment engine cannot reliably assess whether the input image was captured by a camera, but it can assess whether the image resembles an image captured by a camera. By utilizing this characteristic, it is possible to input a high-resolution image generated by a high-resolution engine into the authenticity assessment engine and evaluate whether the high-resolution image generated by the high-resolution engine has been sufficiently enhanced.

[0055] Furthermore, the efficiency and accuracy of both engines may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine in cooperation. In this case, first, the machine learning model of the image quality improvement engine is trained so that when the authenticity evaluation engine evaluates a high-quality image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image acquired by an imaging device, an authenticity label is output. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

[0056] First Embodiment The medical image processing apparatus according to the first embodiment will be described below with reference to Figures 4 and 5. Figure 4 shows an example of the schematic configuration of the image processing apparatus according to this embodiment.

[0057] The image processing device 400 is connected to the photographing device 10 and the display unit 20 via a circuit or a network. Alternatively, the photographing device 10 and the display unit 20 may be directly connected. Although these devices are separate devices in this embodiment, some or all of these devices may be configured as an integrated device. Alternatively, these devices may be connected to any other device via a circuit or a network, or may be configured as an integrated device with any other device.

[0058] The image processing device 400 includes an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). The image processing device 400 may be configured with a plurality of devices each equipped with some of these components. The acquisition unit 401 can acquire various data and images from the imaging device 10 or other devices, and can acquire input from the examiner via an input device (not shown). The input device may be a mouse, keyboard, touch panel, or any other input device. The display unit 20 may be configured as a touch panel display.

[0059] The imaging condition acquisition unit 402 acquires the imaging conditions of the medical image (input image) acquired by the acquisition unit 401. Specifically, the imaging condition group stored in the data structure constituting the medical image is acquired according to the data format of the medical image. Note that if the imaging conditions are not stored in the medical image, the imaging information group including the imaging condition group can be acquired via the acquisition unit 401 from the imaging device 10 or the image management system.

[0060] The image quality improvement possibility determination unit 403 determines whether the medical image can be handled by the image quality improvement unit 404 using the group of imaging conditions acquired by the imaging condition acquisition unit 402. The image quality improvement unit 404 improves the image quality of the medical image that can be handled, and generates a high-quality image suitable for image diagnosis. The output unit 405 displays the high-quality image generated by the image quality improvement unit 404, the input image, various information, etc. on the display unit 20. The output unit 405 may also store the generated high-quality image, etc. in a storage device (storage unit) connected to the image processing device 400.

[0061] Next, a detailed description will be given of the image quality improvement unit 404. The image quality improvement unit 404 is provided with an image quality improvement engine. The image quality improvement method provided by the image quality improvement engine according to this embodiment performs processing using a machine learning algorithm.

[0062] In this embodiment, training of a machine learning model related to a machine learning algorithm uses training data consisting of pairs of input data, which are low-quality images under specific imaging conditions assumed to be processed, and output data, which are high-quality images corresponding to the input data. Note that the specific imaging conditions specifically include a predetermined imaging region, imaging method, imaging angle of view, image size, etc.

[0063] In this embodiment, the input data of the training data is a low-quality image acquired using the same model and settings as the imaging device 10. The output data of the training data is a high-quality image acquired using the same settings and image processing as the imaging device 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing, such as averaging, on a group of images (original images) acquired by multiple captures. Here, high-quality images and low-quality images will be described using OCTA motion contrast data as an example. Here, motion contrast data is data used in OCTA and the like, which is obtained by repeatedly capturing images of the same location on the subject and detecting changes in the subject over time between captures. In this case, a front image can be generated using data from a desired depth range of the calculated motion contrast data (an example of three-dimensional medical image data) of the subject, thereby generating an En-Face image (motion contrast front image) of OCTA. Hereinafter, repeatedly capturing OCT data at the same location is referred to as NOR (Number Of Repeat).

[0064] In this embodiment, two different methods for generating high-quality images and low-quality images by overlay processing will be described with reference to FIG.

[0065] The first method will be described with reference to FIG. 28(a) as an example of a high-quality image, in which motion contrast data is generated from OCT data obtained by repeatedly capturing the same location on a subject. In FIG. 28(a), Im2810 represents three-dimensional motion contrast data, and Im2811 represents two-dimensional motion contrast data constituting the three-dimensional motion contrast data. Im2811-1 to Im2811-3 represent OCT tomographic images (B-scans) used to generate Im2811. Here, NOR refers to the number of OCT tomographic images Im2811-1 to Im2811-3 in FIG. 28(a), and NOR is 3 in the example shown. Im2811-1 to Im2811-3 are captured at a predetermined time interval (Δt). Note that the same location refers to one line in the front direction (XY) of the subject's eye, which corresponds to the location Im2811 in FIG. 28(a). The frontal direction is an example of a direction intersecting the depth direction. Because motion contrast data is data that detects temporal changes, generating this data requires at least two NORs. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, two data are generated when motion contrast data is generated using only OCT data from adjacent time intervals (the first and second, or the second and third), respectively. When motion contrast data is generated using OCT data from distant time intervals (the first and third), a total of three data are generated. In other words, increasing NOR to three, four, etc. increases the number of motion contrast data for the same location. High-quality motion contrast data can be generated by aligning multiple motion contrast data obtained by repeatedly capturing the same location and performing superposition processing such as averaging. Therefore, NOR should be at least three, and preferably five or more. On the other hand, an example of a corresponding low-quality image is motion contrast data before superposition processing such as averaging. In this case, it is desirable to use the low-quality image as a reference image when performing superimposition processing such as averaging.When performing the overlay process, if the position and shape of the target image are transformed relative to the reference image to align them, there is almost no spatial misalignment between the reference image and the image after overlay processing. This makes it easy to pair low-quality and high-quality images. Note that the target image, which has undergone image transformation processing for alignment, can also be used as the low-quality image, rather than the reference image. Multiple pairs can be generated by using each of the original images (reference image and target image) as input data and the corresponding overlaid image as output data. For example, to obtain one overlaid image from a set of 15 original images, a pair of the first original image from the set of original images and the overlaid image, and a pair of the second original image from the set of original images and the overlaid image can be generated. In this way, to obtain one overlaid image from a set of 15 original images, 15 pairs of one image from the set of original images and the overlaid image can be generated. Note that three-dimensional high-quality data can be generated by repeatedly capturing images of the same location in the main scanning (X) direction and then scanning them while shifting them in the sub-scanning (Y) direction.

[0066] The second method, which involves generating a high-quality image by overlaying motion contrast data from multiple images of the same region of the subject, is described with reference to Figure 28(b). The same region refers to an area, such as a 3 x 3 mm or 10 x 10 mm area in the frontal (XY) direction of the subject's eye, and involves acquiring three-dimensional motion contrast data, including the depth direction of the tomographic images. When overlaying the same region by multiple images, it is desirable to perform NOR two or three times to shorten each image capture. Furthermore, to generate high-quality three-dimensional motion contrast data, at least two or more sets of three-dimensional data from the same region are acquired. Figure 28(b) shows an example of multiple sets of three-dimensional motion contrast data. Im2820 to Im2840 are three-dimensional motion contrast data, similar to those described in Figure 28(a). These two or more sets of three-dimensional motion contrast data are used to align the frontal (XY) and depth directions (Z), remove artifacts from each data, and then average the data. This makes it possible to generate a single piece of high-quality 3D motion contrast data with artifacts removed. Generating an arbitrary plane from the 3D motion contrast data results in a high-quality image. Meanwhile, the corresponding low-quality image is preferably an arbitrary plane generated from reference data when performing overlay processing such as averaging. As explained in the first method, there is almost no spatial misalignment between the reference image and the image after averaging, so a low-quality image and a high-quality image can easily be paired. Note that the low-quality image may also be an arbitrary plane generated from target data that has undergone image deformation processing for alignment, rather than the reference data.

[0067] The first method places less strain on the subject because the scan itself is completed in one go. However, the more NORs there are, the longer each scan takes. Also, if artifacts such as eye clouding or eyelashes appear during the scan, it is not always possible to obtain a good image. The second method places a slight burden on the subject because multiple scans are taken. However, the scan time is shorter, and even if an artifact appears in one scan, as long as the artifact is not visible in another scan, a clean image with fewer artifacts can ultimately be obtained. Taking these features into consideration, when collecting data, a method can be chosen that suits the subject's situation.

[0068] In this embodiment, the motion contrast data has been described as an example, but this is not limiting. Because OCT data is captured to generate motion contrast data, the same method can be used with OCT data as described above. Furthermore, although the description of tracking processing has been omitted in this embodiment, it is desirable to capture images while tracking the subject's eye in order to capture images of the same location or region of the subject's eye.

[0069] In this embodiment, a pair of three-dimensional high-resolution data and low-resolution data is created, and any pair of two-dimensional images can be generated from these. This will be described with reference to FIG. 29. For example, if the target image is an OCTA En-Face image, the OCTA En-Face image is generated from the three-dimensional data within a desired depth range. The desired depth range refers to the range in the Z direction in FIG. 28. An example of the OCTA En-Face image generated here is shown in FIG. 29(a). OCTA En-Face images are generated for different depth ranges, such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940), for learning. Note that the types of OCTA En-Face images are not limited to these, and the number of types can be increased by generating OCTA En-Face images with different depth ranges set by changing the reference layer and offset value. During training, OCTA En-Face images of different depths may be trained separately, multiple images of different depth ranges may be combined (e.g., divided into superficial and deep layers), or OCTA En-Face images of all depth ranges may be trained together. For luminance En-Face images generated from OCT data, training is performed using multiple En-Face images generated from any depth range, as with OCTA En-Face. For example, consider a case where the image quality improvement engine includes a machine learning engine obtained using training data containing multiple motion-contrast front images corresponding to different depth ranges of the subject's eye. In this case, the acquisition unit can acquire, as the first image, a motion-contrast front image corresponding to a partial depth range within a long depth range that includes different depth ranges. That is, a motion-contrast front image corresponding to a depth range different from the multiple depth ranges corresponding to the multiple motion-contrast front images included in the training data can be used as the input image for image quality improvement. Of course, a motion-contrast front image of the same depth range as used during training may also be used as the input image for image quality improvement. Furthermore, some depth ranges may be set in response to the examiner pressing an arbitrary button on the user interface, or may be set automatically.The above-described content is not limited to motion contrast front images, but can also be applied to, for example, luminance En-Face images.

[0070] When the image to be processed is a tomographic image, learning is performed using an OCT tomographic image, which is a B-scan, or a tomographic image of motion contrast data. This will be explained using FIG. 29(b). In FIG. 29(b), Im2951 to Im2953 are OCT tomographic images. The images in FIG. 29(b) are different because they represent tomographic images of different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without regard to the difference in position in the sub-scanning direction. However, in the case of images captured at different locations (e.g., the center of the macula, the center of the optic disc), learning may be performed separately for each location, or learning may be performed together without regard to the captured location. Note that the image features of OCT tomographic images and tomographic images of motion contrast data are significantly different, so it is better to learn them separately.

[0071] The superimposed image obtained by the superimposition process emphasizes pixels commonly depicted in the original images, resulting in a high-quality image suitable for image diagnosis. In this case, the generated high-quality image has a high contrast, with a clear distinction between low-brightness and high-brightness areas, as a result of the emphasis on commonly depicted pixels. Furthermore, for example, the superimposed image can reduce random noise that occurs with each capture, and areas that were not well depicted in the original images at a certain point in time can be interpolated using other original images.

[0072] Furthermore, if the input data for a machine learning model needs to consist of multiple images, the necessary number of original image groups can be selected from the original image group and used as input data. For example, if one overlaid image is to be obtained from a group of 15 original images, and two images are required as input data for the machine learning model, 105 (15C2=105) pair groups can be generated.

[0073] It should be noted that pairs that do not contribute to improving image quality can be removed from the training data. For example, if the high-quality images that are output data that make up the training data pairs have image quality that is unsuitable for diagnostic imaging, the images output by the image quality improvement engine that has learned using the training data may also have image quality that is unsuitable for diagnostic imaging. Therefore, by removing pairs whose output data has image quality that is unsuitable for diagnostic imaging from the training data, it is possible to reduce the possibility that the image quality improvement engine will generate images with image quality that is unsuitable for diagnostic imaging.

[0074] Furthermore, if the average luminance or luminance distribution of a pair of images differs significantly, the image quality improvement engine that has trained using the training data may output an image that is unsuitable for image diagnosis, with a luminance distribution that differs significantly from that of the low-quality image. For this reason, pairs of input data and output data with significantly different average luminance or luminance distribution can be removed from the training data.

[0075] Furthermore, if the structure or position of the object depicted in the paired images differs significantly, the image quality improvement engine that has learned using the training data may output an image that is unsuitable for image diagnosis, depicting the object in a structure or position that differs significantly from that of the low-quality image. For this reason, pairs of input data and output data in which the depicted object has a significantly different structure or position can be removed from the training data. Furthermore, from the perspective of maintaining quality, the image quality improvement engine can be configured not to use the high-quality images it outputs as training data.

[0076] By using an image quality improvement engine that has performed machine learning in this way, when a medical image acquired in a single shot is input, the image quality improvement unit 404 can output a high-quality image in which contrast has been increased, noise has been reduced, etc. by overlay processing. Therefore, the image quality improvement unit 404 can generate a high-quality image suitable for image diagnosis based on a low-quality image that is the input image.

[0077] Next, a series of image processing steps according to this embodiment will be described with reference to the flow diagram of Fig. 5. Fig. 5 is a flow diagram of a series of image processing steps according to this embodiment. First, when a series of image processing steps according to this embodiment is started, the process proceeds to step S510.

[0078] In step S510, the acquisition unit 401 acquires, as an input image, an image captured by the image capturing device 10 from the image capturing device 10 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the image capturing device 10. Such a request may be issued, for example, when the image capturing device 10 generates an image, before or after the image generated by the image capturing device 10 is saved in a storage device provided in the image capturing device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0079] The acquisition unit 401 may acquire data for generating an image from the image capturing device 10, and may acquire an image generated by the image processing device 400 based on the data as the input image. In this case, any existing image generation method may be adopted as the image generation method for the image processing device 400 to generate various images.

[0080] In step S520, the photographing condition acquisition unit 402 acquires a group of photographing conditions for the input image. Specifically, the photographing condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. As described above, if the photographing conditions are not stored in the input image, the photographing condition acquisition unit 402 can acquire a photographing information group including the photographing condition group from the photographing device 10 or an image management system (not shown).

[0081] In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether the input image can be improved in image quality by the image quality improvement engine provided in the image quality improvement unit 404. Specifically, the image quality improvement possibility determination unit 403 determines whether the shooting part, shooting method, shooting angle of view, and image size of the input image match the conditions that can be handled by the image quality improvement engine.

[0082] If the image quality improvement possibility determination unit 403 determines all the shooting conditions and determines that they can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550.

[0083] Depending on the settings and implementation of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging region, imaging method, imaging angle of view, and image size. For example, such processing may be performed when the image quality improvement engine is assumed to be capable of comprehensively handling all imaging regions of the subject and is implemented to be able to handle even unknown imaging regions included in the input data. Furthermore, the image quality improvement feasibility determination unit 403 may determine, depending on the desired configuration, whether or not at least one of the imaging region, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the image quality improvement engine.

[0084] In step S540, the image quality improvement unit 404 uses an image quality improvement engine to improve the image quality of the input image and generate a high-quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, which generates a high-quality image with improved image quality. The image quality improvement engine generates a high-quality image that looks like it has been overlaid using the input image, based on a machine learning model that has performed machine learning using training data. Therefore, the image quality improvement engine can generate a high-quality image that has reduced noise and enhanced contrast compared to the input image.

[0085] Depending on the settings and implementation form of the image processing device 400, the image quality improvement unit 404 may input parameters to the image quality improvement engine together with the input image according to the group of imaging conditions, and adjust the degree of image quality improvement, etc. Also, the image quality improvement unit 404 may input parameters according to the examiner's input to the image quality improvement engine together with the input image, and adjust the degree of image quality improvement, etc.

[0086] In step S550, if a high-quality image was generated in step S540, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it was determined in step S530 that high-quality image processing is not possible, the output unit 405 outputs the input image and displays it on the display unit 20. Note that instead of displaying the output image on the display unit 20, the output unit 405 may cause the image capture device 10 or another device to display or store the output image. Furthermore, depending on the settings and implementation form of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image capture device 10 or another device, or convert the data format so that it can be transmitted to an image management system, etc.

[0087] As described above, the image processing device 400 according to this embodiment includes an acquisition unit 401 and an image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) that is an image of a predetermined part of a subject. The image quality improvement unit 404 uses an image quality improvement engine including a machine learning engine to generate a high-quality image (second image) from the input image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image. The image quality improvement engine includes a machine learning engine that uses images obtained by the overlay process as learning data.

[0088] With this configuration, the image processing device 400 according to this embodiment can output a high-quality image in which noise has been reduced and contrast has been enhanced from an input image. As a result, the image processing device 400 can obtain images suitable for image diagnosis, such as clearer images and images in which the area or lesion to be observed is emphasized, without increasing invasiveness or labor for the photographer or subject, and at a lower cost than conventional methods.

[0089] The image processing device 400 further includes an image quality improvement possibility determination unit 403 that determines whether a high-quality image can be generated for an input image using an image quality improvement engine. The image quality improvement possibility determination unit 403 makes this determination based on at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image.

[0090] With this configuration, the image processing device 400 according to this embodiment can exclude input images that cannot be processed by the image quality improvement unit 404 from the image quality improvement processing, thereby reducing the processing load on the image processing device 400 and the occurrence of errors.

[0091] In this embodiment, the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20, but the operation of the output unit 405 is not limited to this. For example, the output unit 405 can also output the high-quality image to the imaging device 10 or other devices connected to the image processing device 400. Therefore, the high-quality image can be displayed on the user interface of these devices, saved in any storage device, used for any image analysis, or sent to an image management system.

[0092] In this embodiment, the image quality improvement possibility determination unit 403 determines whether an input image can be improved in image quality by the image quality improvement engine, and if the input image can be improved in image quality, the image quality improvement unit 404 improves the image quality. On the other hand, if the image capturing device 10 captures images only under shooting conditions that allow for high image quality improvement, the image quality of the images acquired from the image capturing device 10 may be improved unconditionally. In this case, as shown in Fig. 6, the processes of steps S520 and S530 can be omitted, and step S540 can be performed after step S510.

[0093] In this embodiment, the output unit 405 is configured to display a high-quality image on the display unit 20. However, the output unit 405 may also display a high-quality image on the display unit 20 in response to an instruction from the examiner. For example, the output unit 405 may display a high-quality image on the display unit 20 in response to the examiner pressing an arbitrary button on the user interface of the display unit 20. In this case, the output unit 405 may display the high-quality image in place of the input image, or may display the high-quality image alongside the input image.

[0094] Furthermore, when displaying a high-quality image on the display unit 20, the output unit 405 may also display, together with the high-quality image, a display indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. In this case, the display allows the user to easily distinguish that the displayed high-quality image is not the actual image acquired by photography, thereby reducing misdiagnosis and improving diagnostic efficiency. Note that the display indicating that the high-quality image is generated by processing using a machine learning algorithm may be in any form as long as it allows the user to distinguish between the input image and the high-quality image generated by the processing.

[0095] Furthermore, the output unit 405 may display, on the display unit 20, a display indicating what kind of training data the machine learning algorithm used to learn, in addition to a display indicating that the image is a high-quality image generated by processing using a machine learning algorithm. Such a display may include an explanation of the types of input data and output data of the training data, or any display related to the training data, such as the imaging site included in the input data and output data.

[0096] In the image quality improvement engine according to this embodiment, a superimposed image is used as output data of the training data, but the training data is not limited to this. As output data of the training data, a high-quality image obtained by performing at least one of a superimposition process, a group of processes described below, and an imaging method described below, which are means for obtaining a high-quality image, may also be used.

[0097] For example, high-quality images obtained by performing maximum a posteriori probability estimation (MAP) on a group of original images may be used as training data output data. In MAP estimation, a likelihood function is calculated from the probability density of each pixel value in multiple low-quality images, and the calculated likelihood function is used to estimate the true signal value (pixel value).

[0098] The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values ​​that are close to the true signal values. Furthermore, because the estimated signal values ​​are calculated based on probability density, randomly occurring noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the image quality improvement engine can generate a high-quality image suitable for image diagnosis, with reduced noise and high contrast, from the input image. The method for generating pairs of input data and output data for the training data may be the same as when an overlaid image is used as training data.

[0099] Furthermore, a high-quality image obtained by applying a smoothing filter process to an original image may be used as output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which random noise has been reduced from the input image. Furthermore, an image obtained by applying a gradation conversion process to an original image may be used as output data of the training data. In this case, the image quality improvement engine can generate a high-quality image in which contrast has been enhanced from the input image. The method for generating pairs of input data and output data for the training data may be the same as when the superimposed image is used as the training data.

[0100] The input data of the training data may be an image acquired from an imaging device having the same image quality tendency as the imaging device 10. The output data of the training data may be a high-quality image obtained by expensive processing such as an iterative approximation method, or may be a high-quality image acquired by imaging a subject corresponding to the input data with an imaging device with higher performance than the imaging device 10. Furthermore, the output data may be a high-quality image obtained by performing rule-based noise reduction processing. Here, the noise reduction processing may include, for example, replacing a single high-brightness pixel that appears in a low-brightness area and is clearly noise with the average value of nearby low-brightness pixels. Therefore, the image quality improvement engine may use, as training data, images captured by a higher-performance imaging device than the imaging device used to capture the input image, or images acquired through an imaging process involving more steps than the imaging process for the input image. For example, when a motion contrast front image is used as the input image, the image quality improvement engine may use, as training data, images obtained by OCTA imaging using a higher-performance OCT imaging device than the OCT imaging device used to capture the input image, or images acquired through an OCTA imaging process involving more steps than the OCTA imaging process for the input image.

[0101] Although not described in the present embodiment, a high-quality image generated from a plurality of images and used as output data for training data can be generated from a plurality of aligned images. The alignment process may involve, for example, selecting one of the images as a template, determining the similarity with other images while changing the position and angle of the template, determining the amount of misalignment with the template, and correcting each image based on the amount of misalignment. Any other existing alignment process may also be performed.

[0102] When aligning a three-dimensional image, the three-dimensional image may be decomposed into a plurality of two-dimensional images, and the two-dimensional images may be aligned and integrated together to align the three-dimensional images. Alternatively, the two-dimensional image may be decomposed into one-dimensional images, and the two-dimensional images may be aligned and integrated to align the two-dimensional images. Note that these alignments may be performed on data for generating the images, rather than on the images themselves.

[0103] Furthermore, in this embodiment, when the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement unit 404, the process proceeds to step S540, and the image quality improvement process by the image quality improvement unit 404 is started. In response to this, the output unit 405 may display the determination result by the image quality improvement possibility determination unit 403 on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner. At this time, the output unit 405 may display the input image and imaging conditions such as the imaging region acquired for the input image, along with the determination result, on the display unit 20. In this case, the image quality improvement process is performed after the examiner determines whether the determination result is correct, and therefore it is possible to reduce image quality improvement process based on an erroneous determination.

[0104] Alternatively, instead of making a judgment by the image quality improvement possibility judgment unit 403, the output unit 405 may display the input image and the imaging conditions such as the imaging area acquired for the input image on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to instructions from the examiner.

[0105] <Second embodiment> Next, an image processing apparatus according to a second embodiment will be described with reference to FIGS. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in this embodiment, the image quality improvement unit includes multiple image quality improvement engines that perform machine learning using different training data, and generates multiple high-quality images from an input image.

[0106] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0107] The image quality improving unit 404 according to this embodiment is equipped with two or more image quality improving engines, each of which has undergone machine learning using different training data. A method for creating a training data group according to this embodiment will now be described. Specifically, first, a group of pairs of original images as input data and superimposed images as output data, in which various imaging regions are imaged, is prepared. Next, the pair groups are grouped for each imaging region to create a training data group. For example, a training data group is created in such a way that first training data is made up of a group of pairs acquired by imaging a first imaging region, and second training data is made up of a group of pairs acquired by imaging a second imaging region.

[0108] Then, separate image quality improvement engines are used to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, a second image quality improvement engine corresponding to a machine learning model trained with the second training data, and so on.

[0109] Since each of these image quality improvement engines uses different teacher data for training the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the imaging conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine improves the image quality to a high degree for an input image acquired by imaging a first imaging region, and improves the image quality to a low degree for an image acquired by imaging a second imaging region. Similarly, the second image quality improvement engine improves the image quality to a high degree for an input image acquired by imaging a second imaging region, and improves the image quality to a low degree for an image acquired by imaging the first imaging region.

[0110] Since each training data set is composed of pairs grouped by imaging location, the image groups constituting the pair groups have similar image quality trends. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, as long as the imaging location corresponds. Note that the imaging conditions for grouping pairs of training data are not limited to imaging location, but may also be imaging angle of view, image resolution, or a combination of two or more of these.

[0111] A series of image processing steps according to this embodiment will be described below with reference to Fig. 7. Fig. 7 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S710 and S720 are the same as steps S510 and S520 according to the first embodiment, and therefore their description will be omitted. Note that if the image quality of the input image is to be unconditionally improved, after the processing step S720, the processing step S730 may be omitted and the processing may proceed to step S740.

[0112] When the shooting conditions of the input image are acquired in step S720, the process proceeds to step S730. In step S730, the image quality improvement possibility determination unit 403 uses the group of shooting conditions acquired in step S720 to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

[0113] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S760. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S740. Note that, depending on the settings and implementation form of the image processing device 400, step S740 may be performed, as in the first embodiment, even if it is determined that some of the shooting conditions cannot be handled by the image quality improvement engines.

[0114] In step S740, the image quality improvement unit 404 selects an image quality improvement engine from the group of image quality improvement engines to perform image quality improvement processing based on the imaging conditions of the input image acquired in step S720 and the information on the teacher data of the group of image quality improvement engines. Specifically, for example, for an imaging region in the group of imaging conditions acquired in step S720, an image quality improvement engine that has information on teacher data related to the imaging region or surrounding imaging regions and provides a high level of image quality improvement is selected. In the above example, when the imaging region is the first imaging region, the image quality improvement unit 404 selects the first image quality improvement engine.

[0115] In step S750, image quality improvement unit 404 generates a high-quality image by improving the image quality of the input image using the image quality improvement engine selected in step S740. Thereafter, in step S760, if a high-quality image was generated in step S750, output unit 405 outputs the high-quality image and displays it on display unit 20. On the other hand, if image quality improvement processing was determined to be impossible in step S730, output unit 405 outputs the input image and displays it on display unit 20. Note that when displaying the high-quality image on display unit 20, output unit 405 may also display that the high-quality image was generated using the image quality improvement engine selected by image quality improvement unit 404.

[0116] As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine that corresponds to at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution of the input image.

[0117] With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

[0118] In this embodiment, the image quality improvement unit 404 selects an image quality improvement engine to be used for image quality improvement processing based on the shooting conditions of the input image, but the image quality improvement engine selection process is not limited to this. For example, the output unit 405 may display the shooting conditions of the acquired input image and a group of image quality improvement engines on the user interface of the display unit 20, and the image quality improvement unit 404 may select an image quality improvement engine to be used for image quality improvement processing in response to an instruction from the examiner. The output unit 405 may also display information about the training data used to train each image quality improvement engine on the display unit 20 along with the group of image quality improvement engines. The display format of the information about the training data used to train the image quality improvement engines may be arbitrary. For example, the group of image quality improvement engines may be displayed using names related to the training data used for training.

[0119] Furthermore, the output unit 405 may display the image quality improvement engine selected by the image quality improvement unit 404 on the user interface of the display unit 20 and receive instructions from the examiner. In this case, the image quality improvement unit 404 may determine, in response to instructions from the examiner, whether or not to finally select the image quality improvement engine as the image quality improvement engine to be used in the image quality improvement process.

[0120] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0121] <Third embodiment> Next, an image processing apparatus according to a third embodiment will be described with reference to FIGS. In the first and second embodiments, the imaging condition acquisition unit 402 acquires an imaging condition group from the data structure of the input image, etc. In contrast, in the present embodiment, the imaging condition acquisition unit estimates the imaging part or imaging area of ​​the input image based on the input image using an imaging location estimation engine.

[0122] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the second embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the second embodiment. The configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first and second embodiments, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0123] The imaging condition acquisition unit 402 according to this embodiment is provided with an imaging location estimation engine that estimates the imaging part or imaging area depicted in the input image acquired by the acquisition unit 401. The imaging location estimation method provided in the imaging location estimation engine according to this embodiment performs estimation processing using a machine learning algorithm.

[0124] In this embodiment, training of a machine learning model for an imaging location estimation method using a machine learning algorithm uses training data consisting of pairs of input data, which are images, and output data, which are imaging site labels or imaging area labels corresponding to the input data. Here, the input data refers to an image with specific imaging conditions that is expected to be processed (input image). The input data is preferably an image acquired from an imaging device with the same image quality tendencies as the imaging device 10, and even better, the same model with the same settings as the imaging device 10. The types of imaging site labels and imaging area labels that are output data may be imaging sites or imaging areas at least partially included in the input data. For example, in the case of OCT, the types of imaging site labels that are output data may be "macula," "optic disc," "macula and optic disc," "other," etc.

[0125] The imaging location estimation engine according to this embodiment has learned using such training data, and is thereby able to output the location of the imaging body part or imaging area depicted in the input image. Furthermore, the imaging location estimation engine can also output the probability of the imaging body part or imaging area being the imaging body part or area for each imaging body part label or imaging area label at a required level of detail. By using the imaging location estimation engine, the imaging condition acquisition unit 402 can estimate the imaging body part or imaging area of ​​the input image based on the input image and acquire the estimation results as imaging conditions for the input image. When the imaging location estimation engine outputs the probability of the imaging body part or imaging area being the imaging body part or imaging area for each imaging body part label or imaging area label, the imaging condition acquisition unit 402 acquires the imaging body part or imaging area with the highest probability as the imaging conditions for the input image.

[0126] Next, as in the second embodiment, a series of image processing steps according to this embodiment will be described with reference to the flow diagram of Fig. 7. Note that the processing steps of step S710 and steps S730 to S760 according to this embodiment are similar to those in the second embodiment, and therefore description thereof will be omitted. Note that if the image quality of the input image is to be unconditionally improved, after the processing step S720, the processing step S730 may be omitted and the processing may proceed to step S740.

[0127] When the input image is acquired in step S710, the process proceeds to step S720. In step S720, the photographing condition acquisition unit 402 acquires a group of photographing conditions for the input image acquired in step S710.

[0128] Specifically, the imaging condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Also, if the imaging condition group does not include information on the imaging part or imaging area, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine and estimates which imaging part the input image was acquired by imaging. Specifically, the imaging condition acquisition unit 402 inputs the input image to the imaging location estimation engine, evaluates the output probability for each imaging part label group, and sets and acquires the imaging part with the highest probability as the imaging condition for the input image.

[0129] In addition, if the input image does not store any shooting conditions other than the shooting part and shooting area, the shooting condition acquisition unit 402 can acquire a group of shooting information including a group of shooting conditions from the shooting device 10 or an image management system not shown.

[0130] The subsequent processing is the same as the series of image processing according to the second embodiment, and therefore a description thereof will be omitted.

[0131] As described above, the photographing condition acquisition unit 402 according to this embodiment functions as an estimation unit that estimates at least one of the photographing part and the photographing area of ​​the input image. The photographing condition acquisition unit 402 includes a photographing location estimation engine that uses images labeled with the photographing part and the photographing area as learning data, and estimates the photographing part and the photographing area of ​​the input image by inputting the input image to the photographing location estimation engine.

[0132] As a result, the image processing device 400 according to this embodiment can acquire the imaging conditions for the imaging part and imaging area of ​​the input image based on the input image.

[0133] In this embodiment, the photographing condition acquisition unit 402 estimates the photographing body part and the photographing area of ​​the input image using the photographing location estimation engine when the photographing condition group does not include information on the photographing body part and the photographing area. However, the situation in which the photographing body part and the photographing area are estimated using the photographing location estimation engine is not limited to this. The photographing condition acquisition unit 402 may estimate the photographing body part and the photographing area using the photographing location estimation engine even when the information on the photographing body part and the photographing area included in the data structure of the input image is insufficient to provide information at the required level of detail.

[0134] Furthermore, regardless of whether the data structure of the input image includes information about the photographing part and the photographing area, the photographing condition acquisition unit 402 may estimate the photographing part and the photographing area of ​​the input image using the photographing location estimation engine. In this case, the output unit 405 may display the estimation result output from the photographing location estimation engine and the information about the photographing part and the photographing area included in the data structure of the input image on the display unit 20, and the photographing condition acquisition unit 402 may determine these photographing conditions in response to an instruction from the examiner.

[0135] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0136] <Fourth embodiment> Next, an image processing device according to a fourth embodiment will be described with reference to Figures 4, 5, 8, and 9. In this embodiment, an image quality improvement unit enlarges or reduces an input image so that the input image has an image size that can be handled by an image quality improvement engine. The image quality improvement unit also reduces or enlarges an output image from the image quality improvement engine so that the image size of the output image matches the image size of the input image, thereby generating a high-quality image.

[0137] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0138] The image quality improvement unit 404 according to this embodiment is provided with an image quality improvement engine similar to the image quality improvement engine according to the first embodiment. However, in this embodiment, as training data used for learning of the image quality improvement engine, a group of pairs of input data and output data, which are made up of a group of images obtained by enlarging or reducing the input data image and the output data image so that they have a certain image size, is used.

[0139] Now, with reference to Fig. 8, the training data for the image quality improvement engine according to this embodiment will be described. As shown in Fig. 8, for example, consider the case where there is a low-quality image Im810 and a high-quality image Im820 that are smaller than a certain image size set for the training data. In this case, the low-quality image Im810 and the high-quality image Im820 are each enlarged so that they match the certain image size set for the training data. Then, the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are paired, and this pair is used as one piece of training data.

[0140] As in the first embodiment, the input data of the training data uses an image having specific imaging conditions assumed to be the processing target (input image), but the specific imaging conditions are a predetermined imaging region, imaging method, and imaging angle of view. In other words, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include image size.

[0141] The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such training data to improve the image quality of an input image and generate a high-quality image. In this case, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain image size set for the training data, and inputs the deformed image into the image quality improvement engine. The image quality improvement unit 404 also reduces or enlarges the output image from the image quality improvement engine so that it has the image size of the input image, thereby generating a high-quality image. Therefore, the image quality improvement unit 404 according to this embodiment can use the image quality improvement engine to improve the image quality of even input images with image sizes that could not be handled in the first embodiment, and generate a high-quality image.

[0142] Next, a series of image processing steps according to this embodiment will be described with reference to Figures 5 and 9. Figure 9 is a flow diagram of the image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are the same as those in the first embodiment, and therefore their description will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than image size, the processing step S530 may be omitted after the processing step S520, and the processing may proceed to step S540.

[0143] In step S520, as in the first embodiment, once the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging region, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

[0144] If the image quality improvement feasibility determination unit 403 determines the imaging region, imaging method, and imaging angle of view and determines that the input image is manageable, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image is not manageable based on some of the imaging region, imaging method, and imaging angle of view.

[0145] When the process proceeds to step S540, the image quality improvement process according to this embodiment, shown in Fig. 9, begins. In the image quality improvement process according to this embodiment, first, in step S910, image quality improvement unit 404 enlarges or reduces the input image to a fixed image size set for the training data, and generates a deformed image.

[0146] Next, in step S920, the image quality improving unit 404 inputs the generated deformed image to an image quality improving engine to obtain a high-quality deformed image.

[0147] Thereafter, in step S930, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image, thereby generating a high-quality image. Once the image quality improvement unit 404 has generated a high-quality image in step S930, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is the same as step S550 in the first embodiment, and therefore a description thereof will be omitted.

[0148] As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of an input image to an image size that can be handled by the image quality improvement engine, and inputs the adjusted image to the image quality improvement engine. The image quality improvement unit 404 also generates a high-quality image by adjusting the output image from the image quality improvement engine to the original image size of the input image. As a result, the image processing device 400 according to this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images suitable for image diagnosis.

[0149] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0150] <Fifth embodiment> Next, an image processing device according to a fifth embodiment will be described with reference to Figures 4, 5, 10, and 11. In this embodiment, an image quality improvement unit generates a high-quality image through image quality improvement processing using an image quality improvement engine based on a certain resolution.

[0151] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0152] The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the training data used for training the image quality improvement engine is different from the training data in the first embodiment. Specifically, the image group constituting the training data pair group of input data and output data is enlarged or reduced to an image size such that the resolution of the image group becomes a fixed resolution, and then padding is performed to obtain a sufficiently large fixed image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution relative to the imaging area.

[0153] Now, with reference to Fig. 10, the training data for the image quality improvement engine according to this embodiment will be described. As shown in Fig. 10, for example, consider the case where there is a low-quality image Im1010 and a high-quality image Im1020, both of which have a resolution lower than the fixed resolution set for the training data. In this case, the low-quality image Im1010 and the high-quality image Im1020 are each enlarged so that they match the fixed resolution set for the training data. Furthermore, each of the enlarged low-quality image Im1010 and the high-quality image Im1020 is padded so that they match the fixed image size set for the training data. Then, the enlarged and padded low-quality image Im1011 and high-quality image Im1021 are paired, and this pair is used as one piece of training data.

[0154] The fixed image size set for the training data is the maximum possible image size when the image expected to be processed (input image) is enlarged or reduced to a fixed resolution. If the fixed image size is not large enough, the image input to the image quality improvement engine may become too large for the machine learning model to handle.

[0155] In addition, the padded area is filled with a fixed pixel value, filled with a neighboring pixel value, or mirror padded according to the characteristics of the machine learning model so as to effectively improve image quality. As in the first embodiment, an image having specific imaging conditions assumed to be the processing target is used as input data, and the specific imaging conditions are a predetermined imaging region, imaging method, and imaging angle of view. In other words, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include image size.

[0156] The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such training data to improve the image quality of an input image and generate a high-quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain resolution set for the training data. The image quality improvement unit 404 also pads the deformed image so that it has a certain image size set for the training data, generates a padded image, and inputs the padded image to the image quality improvement engine.

[0157] Furthermore, the image quality improving unit 404 trims the high-quality padded image output from the image quality improving engine by the amount of the padded area, and generates a high-quality transformed image. Thereafter, the image quality improving unit 404 reduces or enlarges the generated high-quality transformed image so that it matches the image size of the input image, thereby generating a high-quality image.

[0158] Therefore, the image quality improvement unit 404 according to this embodiment can improve the image quality of an input image of an image size that could not be handled in the first embodiment by using an image quality improvement engine to generate a high-quality image.

[0159] Next, a series of image processing steps according to this embodiment will be described with reference to FIGS. 11 is a flow diagram of the image quality improvement process according to this embodiment. Note that the processes of steps S510, S520, and S550 according to this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than image size, after the process of step S520, the process of step S530 may be omitted and the process may proceed to step S540.

[0160] In step S520, as in the first embodiment, once the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging region, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

[0161] If the image quality improvement feasibility determination unit 403 determines the imaging region, imaging method, and imaging angle of view and determines that the input image is manageable, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image is not manageable based on some of the imaging region, imaging method, and imaging angle of view.

[0162] When the process proceeds to step S540, the image quality improvement process according to this embodiment starts, as shown in Fig. 11. In the image quality improvement process according to this embodiment, first, in step S1110, image quality improvement unit 404 enlarges or reduces the input image to a certain resolution set for the training data, and generates a deformed image.

[0163] Next, in step S1120, the image quality improvement unit 404 performs padding on the generated transformed image so that the image size is the same as the image size set for the training data, thereby generating a padded image. At this time, the image quality improvement unit 404 fills the padded area with a fixed pixel value, fills with a neighboring pixel value, or performs mirror padding in accordance with the characteristics of the machine learning model so that the image quality can be effectively improved.

[0164] In step S1130, the image quality improving unit 404 inputs the padding image to the image quality improving engine and obtains a high-quality padding image. Next, in step S1140, the image quality improving unit 404 trims the high-quality padded image by the area padded in step S1120, and generates a high-quality transformed image.

[0165] Thereafter, in step S1150, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image, thereby generating a high-quality image. Once the image quality improvement unit 404 generates a high-quality image in step S1130, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is the same as step S550 in the first embodiment, and therefore a description thereof will be omitted.

[0166] As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image so that the resolution of the input image is a predetermined resolution. Furthermore, the image quality improvement unit 404 generates a padded image by padding the input image whose image size has been adjusted so that the adjusted image size is an image size that can be handled by the image quality improvement engine, and inputs the padded image to the image quality improvement engine. The image quality improvement unit 404 then trims the output image from the image quality improvement engine by the amount of the padded area. The image quality improvement unit 404 then generates a high-quality image by adjusting the image size of the trimmed image to the original image size of the input image.

[0167] As a result, the image quality improvement unit 404 of this embodiment can generate a high-quality image by improving the image quality using the image quality improvement engine, even for an input image of an image size that could not be handled in the first embodiment. Furthermore, by using an image quality improvement engine that has learned using training data based on resolution, it may be possible to improve the image quality of an input image more efficiently than the image quality improvement engine of the fourth embodiment, which simply processes images of the same image size.

[0168] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0169] Sixth Embodiment Next, an image processing device according to a sixth embodiment will be described with reference to Figures 4, 5, 12, and 13. In this embodiment, an image quality improving unit improves the image quality of an input image for each region of a certain image size, thereby generating a high-quality image.

[0170] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0171] The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, in this embodiment, the training data used for training the image quality improvement engine is different from the training data of the first embodiment. Specifically, pairs of input data, which are low-quality images, and output data, which are high-quality images, that make up the training data are made up of rectangular area images of a certain image size that correspond in positional relationship in the low-quality image and the high-quality image. Note that the rectangular area is an example of a partial area, and does not have to be rectangular and may have any shape.

[0172] Here, the training data of the image quality improvement engine according to this embodiment will be described with reference to Fig. 12. As shown in Fig. 12, consider a case where one of the pairs constituting the training data includes, for example, an original image Im1210, which is a low-quality image, and an overlaid image Im1220, which is a high-quality image. In this case, in the first embodiment, the input data of the training data is Im1210, and the output data is Im1220.

[0173] In contrast, in this embodiment, a rectangular area image R1211 of an original image Im1210 is used as input data, and a rectangular area image R1221 of a superimposed image Im1220, which is the same shooting area as the rectangular area image R1211, is used as output data. The rectangular area image R1211, which is the input data, and the rectangular area image R1221, which is the output data, form a pair of training data (hereinafter, a first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images of a fixed image size. The original image Im1210 and the superimposed image Im1220 may be aligned by any method. The corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be determined by any method, such as template matching. Depending on the design of the image quality improvement engine, the input data and the output data may have different image sizes and numbers of dimensions. For example, when the processing target is an OCT image, if the input data is a part of a B-scan image (two-dimensional image), the output data may be a part of an A-scan image (one-dimensional image).

[0174] The fixed image size for the rectangular area images R1211 and R1221 can be determined, for example, from the common divisor of the pixel counts for each dimension of a group of image sizes of images expected to be processed (input images). In this case, it is possible to prevent overlapping of the positional relationships of the group of rectangular area images output by the image quality improvement engine. Specifically, for example, consider a case where the image expected to be processed is a two-dimensional image, and a first image size among the group of image sizes is 500 pixels wide and 500 pixels high, and a second image size is 100 pixels wide and 100 pixels high. Here, the fixed image size for the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, the fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, 25 pixels wide and 25 pixels high, etc.

[0175] If the image to be processed is three-dimensional, the number of pixels is determined for width, height, and depth. Note that multiple rectangular areas can be set for each pair of a low-quality image corresponding to the input data and a high-quality image corresponding to the output data. For example, rectangular area image R1212 from original image Im1210 is used as input data, and rectangular area image R1222 from superimposed image Im1220, which is the same shooting area as rectangular area image R1212, is used as output data. Then, a pair of training data is formed from rectangular area image R1212, which is the input data, and rectangular area image R1222, which is the output data. This allows a rectangular area image pair other than the first rectangular area image pair to be created.

[0176] In addition, by creating a large number of pairs of rectangular area images while converting rectangular area images into images with different coordinates, the group of pairs constituting the training data can be enriched, and an image quality improvement engine trained using these training pairs can be expected to efficiently improve image quality. However, pairs that do not contribute to improving the image quality of the machine learning model can be excluded from the training data. For example, if a rectangular area image created from a high-quality image, which is output data of a pair, has image quality that is not suitable for diagnosis, the image output by an image quality improvement engine trained using such training data may also have image quality that is not suitable for image diagnosis. Therefore, pairs containing such high-quality images can be removed from the training data.

[0177] Furthermore, for example, if the average luminance or luminance distribution of a pair of rectangular area images created from a low-quality image and a high-quality image differ significantly, such pairs can be removed from the training data. If training is performed using such training data, the image quality improvement engine may output an image that has a luminance distribution that differs significantly from the input image and is unsuitable for image diagnosis.

[0178] Furthermore, consider a case where the structure or position of the object depicted in a pair of rectangular area images created from a low-quality image and a pair of rectangular area images created from a high-quality image are significantly different. In this case, an image quality improvement engine that has learned using such training data may output an image that is unsuitable for image diagnosis, depicting the object in a structure or position significantly different from that of the input image. Therefore, such pairs can be removed from the training data.

[0179] As in the first embodiment, images having specific imaging conditions assumed to be processed are used as input data for the training data, but the specific imaging conditions are predetermined imaging regions, imaging methods, and imaging angle of view. In other words, unlike the first embodiment, the specific imaging conditions according to this embodiment do not include image size.

[0180] The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine trained using such training data to improve the image quality of an input image and generate a high-quality image. In this process, the image quality improvement unit 404 divides the input image into a group of contiguous rectangular area images with a fixed image size set for the training data. The image quality improvement unit 404 improves the image quality of each of the divided rectangular area images using the image quality improvement engine to generate a group of high-quality rectangular area images. The image quality improvement unit 404 then arranges and combines the generated high-quality rectangular area images according to the positional relationship of the input image to generate a high-quality image. During training, as long as the positional relationship between the input data and output data, which are paired images, corresponds, each rectangular area may be clipped (extracted) from any location in the low-quality image and the high-quality image. Meanwhile, during image quality improvement, the input image may be divided into a group of contiguous rectangular area images with no gaps. Furthermore, the image size of each paired image during training and the image size of each rectangular area image during image quality improvement may be set to correspond to each other (e.g., be the same). This makes it possible to improve learning efficiency while preventing unnecessary calculations and problems that can lead to an image not being generated due to missing parts.

[0181] In this way, the image quality improvement unit 404 of this embodiment improves the image quality of the input image in units of rectangular areas and combines the improved image quality images, thereby improving the image quality of images of image sizes that could not be handled in the first embodiment and generating high-quality images.

[0182] Next, a series of image processing steps according to this embodiment will be described with reference to Figures 5, 13, and 14. Figure 13 is a flow diagram of image quality improvement processing according to this embodiment. Note that the processing steps S510, S520, and S550 according to this embodiment are similar to those in the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than image size, the processing step S530 may be omitted after the processing step S520, and the processing may proceed to step S540.

[0183] In step S520, as in the first embodiment, once the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement feasibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement feasibility determination unit 403 determines whether the imaging conditions of the input image are imaging region, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement feasibility determination unit 403 does not determine the image size.

[0184] If the image quality improvement feasibility determination unit 403 determines the imaging region, imaging method, and imaging angle of view and determines that the input image is manageable, the process proceeds to step S540. On the other hand, if the image quality improvement feasibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image is not manageable based on some of the imaging region, imaging method, and imaging angle of view.

[0185] When the process proceeds to step S540, the image quality improvement process according to this embodiment, shown in FIG. 13, begins. This will be described with reference to FIG. 14. In the image quality improvement process according to this embodiment, first, in step S1310, as shown in FIG. 14(a), the input image is divided into a group of continuous rectangular region images with no gaps and a fixed image size (size shown in R1411) set for the training data. Here, FIG. 14(a) shows an example in which the input image Im1410 is divided into a group of rectangular region images R1411 to R1426 with fixed image sizes. Note that, as mentioned above, depending on the design of the image quality improvement engine, the input image and output image of the image quality improvement engine may have different image sizes and numbers of dimensions. In this case, the division positions of the input image can be adjusted by overlapping or separating them so that the combined high-quality image generated in step S1320 has no defects. FIG. 14(b) shows an example in which the division positions are overlapped. In FIG. 14(b), R1411' and R1412' indicate overlapping regions. Although not shown to avoid complexity, R1413 to R1426 also have similar overlapping regions R1413' to R1426'. The rectangular region size set for the training data in the case of FIG. 14(b) is the size shown in R1411'. Since no data exists in the periphery (top, bottom, left, and right edges) of the input image Im1410, the image is filled with a fixed pixel value, filled with neighboring pixel values, or mirror padded. Furthermore, depending on the image quality enhancement engine, filtering may reduce the accuracy of image enhancement in the periphery (top, bottom, left, and right edges) of the image. Therefore, rectangular region images may be set with overlapping division positions as shown in FIG. 14(b), and then a portion of the rectangular region image may be trimmed and combined to form the final image. The size of the rectangular region is set according to the characteristics of the image quality enhancement engine. Although Fig. 14(a) and (b) show examples of OCT tomographic images, the input image (Im1450) can also be a frontal image such as an OCTA En-Face image, as shown in Fig. 14(c) and (d), and similar processing is possible. The size of the rectangular region image is set appropriately depending on the target image and the type of image quality improvement engine.

[0186] Next, in step S1320, the image quality improvement unit 404 improves the image quality of each of the rectangular area images R1411 to R1426, or the rectangular area images R1411' to R1426' if an overlapping area is set, using an image quality improvement engine to generate a group of high-image-quality rectangular area images.

[0187] Then, in step S1330, the image quality improving unit 404 arranges each of the generated high-quality rectangular area images in the same positional relationship as the rectangular area images R1411 to R1426 obtained by dividing the input image, and combines them to generate a high-quality image. If an overlapping area is set, the rectangular area images R1411' to R1426' are arranged in the same positional relationship as the rectangular area images R1411 to R1426, and then the rectangular area images R1411 to R1426 are cut out and combined to generate a high-quality image. Note that the overlapping area may be used to correct the luminance values ​​of the rectangular area images R1411' to R1426'. For example, a reference rectangular area image may be arbitrarily set. Then, by measuring the luminance values ​​of the same coordinate points in adjacent rectangular images that have overlapping areas with the reference rectangular image, the difference (ratio) in luminance values ​​between the adjacent images can be determined. Similarly, by calculating the difference (ratio) in luminance values ​​in the overlapping areas of all images, correction can be performed to eliminate uneven luminance values ​​overall. It is not necessary to use the entire overlapping area for brightness value correction, and a portion of the overlapping area (several peripheral pixels) may not be used.

[0188] As described above, the image quality improving unit 404 according to this embodiment divides an input image into a plurality of rectangular area images (third images) R1411 to R1426 of a predetermined image size. The image quality improving unit 404 then inputs the divided rectangular area images R1411 to R1426 into an image quality improving engine to generate a plurality of fourth images, and then integrates the plurality of fourth images to generate a high-quality image. Note that if rectangular area groups overlap in position during integration, the pixel values ​​of the rectangular area groups can be integrated or overwritten.

[0189] As a result, the image quality improving unit 404 of this embodiment can use the image quality improving engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images. Furthermore, if the training data is created from a plurality of images obtained by dividing low-quality images and high-quality images into predetermined image sizes, a large amount of training data can be created from a small number of images. Therefore, in this case, the number of low-quality images and high-quality images used to create the training data can be reduced.

[0190] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0191] Seventh Embodiment Next, an image processing device according to a seventh embodiment will be described with reference to Figures 15 to 17. In this embodiment, an image quality evaluation unit selects the image with the highest image quality from among multiple high-image-quality images output from multiple image quality improvement engines in response to an instruction from an examiner.

[0192] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0193] 15 shows a schematic configuration of an image processing device 1500 according to this embodiment. The image processing device 1500 according to this embodiment is provided with an image quality evaluation unit 1506 in addition to an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405. Note that the image processing device 1500 may be configured with a plurality of devices each provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, and therefore the same reference numerals as those in FIG. 4 are used to indicate the configuration, and description thereof will be omitted.

[0194] Furthermore, the image processing device 1500 may be connected to the photographing device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. These devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

[0195] The image quality improving unit 404 according to this embodiment is equipped with two or more image quality improving engines, each of which has undergone machine learning using different training data. Here, a method for creating a training data group according to this embodiment will be described. Specifically, first, a group of pairs of input data, which are low-quality images, and output data, which are high-quality images, captured under various imaging conditions are prepared. Next, a training data group is created by grouping the pair groups according to any combination of imaging conditions. For example, a first training data group is created consisting of a group of pairs acquired under a first combination of imaging conditions, and a second training data group is created consisting of a group of pairs acquired under a second combination of imaging conditions.

[0196] Then, each training data is used to perform machine learning in a separate image quality improvement engine. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the first training data.

[0197] Since each of these image quality improvement engines uses different teacher data for training the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine improves the image quality to a high degree for input images acquired by shooting under a first combination of shooting conditions, and improves the image quality to a low degree for images acquired by shooting under a second combination of shooting conditions. Similarly, the second image quality improvement engine improves the image quality to a high degree for input images acquired by shooting under the second shooting conditions, and improves the image quality to a low degree for images acquired by shooting under the first shooting conditions.

[0198] Since each training data set is composed of pairs grouped by a combination of imaging conditions, the image groups constituting the pair groups have similar image quality trends. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, as long as the combination of imaging conditions is compatible. Note that the combination of imaging conditions for grouping pairs of training data may be any combination, and may be, for example, a combination of two or more of the imaging region, imaging angle of view, and image resolution. Furthermore, training data may be grouped based on a single imaging condition, as in the second embodiment.

[0199] The image quality evaluation unit 1506 selects the high-quality image with the highest image quality from the multiple high-quality images generated by the image quality improvement unit 404 using multiple image quality improvement engines, in accordance with an instruction from the examiner.

[0200] The output unit 405 can display the high-quality image selected by the image quality evaluation unit 1506 on the display unit 20 or output it to another device. The output unit 405 can display multiple high-quality images generated by the image quality improvement unit 404 on the display unit 20, and the image quality evaluation unit 1506 can select the high-quality image with the highest image quality in response to an instruction from the examiner who checks the display unit 20.

[0201] As a result, the image processing device 1500 can output the high-quality image with the highest image quality in accordance with the examiner's instructions from among the multiple high-quality images generated using the multiple image quality improvement engines.

[0202] A series of image processing steps according to this embodiment will be described below with reference to Figures 16 and 17. Figure 16 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S1610 and S1620 according to this embodiment are similar to the processing steps S510 and S520 in the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1620, the processing step S1630 may be omitted and the processing may proceed to step S1640.

[0203] In step S1620, as in the first embodiment, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, and then the process proceeds to step S1630. In step S1630, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image, as in the second embodiment.

[0204] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed, as in the first embodiment, even if it is determined that some of the shooting conditions cannot be handled by the image quality improvement engines.

[0205] In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engines, and generates a group of high-quality images.

[0206] In step S1650, the image quality assessment unit 1506 selects the image with the highest image quality from the group of high-image-quality images generated in step S1640. Specifically, first, the output unit 405 causes the group of high-image-quality images generated in step S1640 to be displayed on the user interface of the display unit 20.

[0207] An example of the interface is shown in Fig. 17. The interface displays an input image Im1710 and high-quality images Im1720, Im1730, Im1740, and Im1750 output by each of the image quality improvement engines. The examiner operates an arbitrary input device (not shown) to specify the image with the highest image quality, i.e., the image most suitable for image diagnosis, from the image group (high-quality images Im1720 to Im1750). Note that, since an input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the input image may be added to the image group that is the subject of the examiner's specification.

[0208] Thereafter, the image quality evaluation unit 1506 selects the high-quality image designated by the examiner as the image with the highest quality.

[0209] In step S1660, the output unit 405 displays the image selected in step S1650 on the display unit 20 or outputs it to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if the examiner specifies an input image or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0210] As described above, the image quality improving unit 404 according to this embodiment generates a plurality of high-quality images from an input image using a plurality of image quality improving engines, and the output unit 405 of the image processing device 1500 outputs at least one of the plurality of high-quality images in response to an instruction from the examiner. In particular, in this embodiment, the output unit 405 outputs the image with the highest image quality in response to an instruction from the examiner. As a result, the image processing device 1500 can output a high-quality image having a high image quality according to the examiner's instructions from among a plurality of high-quality images generated using a plurality of image quality improvement engines.

[0211] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0212] Eighth Embodiment Next, an image processing device according to an eighth embodiment will be described with reference to Figures 15 and 16. In this embodiment, an image quality evaluation unit uses an image quality evaluation engine to select the image with the highest image quality from among multiple high-image-quality images output from multiple image quality improvement engines.

[0213] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 1500 according to the seventh embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the seventh embodiment. Note that, since the configuration of the image processing device according to this embodiment is the same as that of the image processing device according to the seventh embodiment, the configuration shown in FIG. 15 will be indicated by the same reference numerals, and description thereof will be omitted.

[0214] The image quality evaluation unit 1506 according to this embodiment is equipped with an image quality evaluation engine that evaluates the image quality of an input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm. The input data pairs that constitute the training data for the machine learning model are image groups consisting of a group of low-quality images and a group of high-quality images captured in advance under various shooting conditions. Furthermore, the output data pairs that constitute the training data for the machine learning model are, for example, image quality evaluation index groups set for each of the image groups of the input data by an examiner performing image diagnosis.

[0215] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 16. Note that the processing steps S1610, S1620, S1630, and S1660 according to this embodiment are similar to those steps in the seventh embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1620, the processing step S1630 may be omitted and the processing may proceed to step S1640.

[0216] In step S1630, as in the seventh embodiment, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, as in the first embodiment, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed even if it is determined that some of the shooting conditions cannot be handled by the image quality improvement engine.

[0217] In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engines, and generates a group of high-quality images.

[0218] In step S1650, the image quality evaluation unit 1506 selects the image with the highest image quality from the group of high-quality images generated in step S1640. Specifically, the image quality evaluation unit 1506 first inputs the group of high-quality images generated in step S1640 to an image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index for each input high-quality image based on learning. The image quality evaluation unit 1506 selects the high-quality image with the highest image quality evaluation index calculated. Note that an input image that has not been improved by the image quality improvement engine may be more suitable for image diagnosis, so the image quality evaluation unit 1506 may also input the input image to the image quality evaluation engine and add the image quality evaluation index for the input image to the selection. Step S1660 is similar to step S1660 in the seventh embodiment, and therefore description thereof will be omitted.

[0219] As described above, the image processing device 1500 according to this embodiment further includes an image quality evaluation unit 1506 that evaluates the image quality of a high-quality image. The image quality improvement unit 404 generates a plurality of high-quality images from an input image using a plurality of image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the plurality of high-quality images according to the evaluation result by the image quality evaluation unit 1506. In particular, the image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that uses evaluation values ​​obtained by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects, from the plurality of high-quality images, the high-quality image that has received the highest evaluation result using the image quality evaluation engine by the image quality evaluation unit 1506. The output unit 405 outputs the high-quality image with the highest evaluation value selected by the image quality evaluation unit 1506.

[0220] As a result, the image processing device 1500 according to this embodiment can easily output the high-quality image most suitable for image diagnosis from among a plurality of high-quality images, based on the output of the image quality evaluation engine.

[0221] In this embodiment, the image quality evaluation unit 1506 selects a high-quality image having the highest image quality evaluation index among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality image on the display unit 20. However, the configuration of the image quality evaluation unit 1506 is not limited to this. For example, the image quality evaluation unit 1506 may select high-quality images having the top several image quality evaluation indices among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 may display the selected high-quality image on the display unit 20. Alternatively, the output unit 405 may display the image quality evaluation indices output by the image quality evaluation engine together with the corresponding high-quality image on the display unit 20, and the image quality evaluation unit 1506 may select the image with the highest image quality in accordance with instructions from the examiner.

[0222] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0223] <Ninth embodiment> Next, an image processing device according to a ninth embodiment will be described with reference to Figures 18 and 19. In this embodiment, an authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

[0224] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0225] 18 shows a schematic configuration of an image processing device 1800 according to this embodiment. The image processing device 1800 according to this embodiment is provided with an authenticity evaluation unit 1807 in addition to an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405. The image processing device 1800 may be configured with a plurality of devices each provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, and therefore the same reference numerals as those in FIG. 4 are used to indicate the configuration, and description thereof will be omitted.

[0226] Furthermore, the image processing device 1800 may be connected to the photographing device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. These devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

[0227] The authenticity evaluation unit 1807 is equipped with an authenticity evaluation engine. The authenticity evaluation unit 1807 uses the authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement engine has been sufficiently improved in quality. The authenticity evaluation processing method in the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

[0228] The training data for training the machine learning model includes a group of pairs of high-quality images taken in advance under various shooting conditions and labels indicating that the images were taken and acquired by the target camera (hereinafter referred to as "genuine labels").The training data also includes a group of pairs of high-quality images generated by inputting low-quality images into a high-quality image enhancement engine with low image enhancement accuracy and labels indicating that the images were not taken and acquired by the target camera (hereinafter referred to as "fake labels").

[0229] An authenticity evaluation engine that has been trained using such training data cannot reliably evaluate whether an input image is an image that was captured and acquired by an imaging device, but it can evaluate whether the image has the appearance of an image that was captured and acquired by an imaging device. By utilizing this characteristic, the authenticity evaluation unit 1807 can input a high-quality image generated by the image quality improvement unit 404 into the authenticity evaluation engine and evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

[0230] If the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved, the output unit 405 displays the high-quality image on the display unit 20. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved, the output unit 405 displays the input image on the display unit 20. When displaying the input image, the output unit 405 can display on the display unit 20 that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved or that the image being displayed is the input image.

[0231] A series of image processing steps according to this embodiment will be described below with reference to Fig. 19. Fig. 19 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S1910 to S1940 according to this embodiment are the same as the processing steps S510 to S540 in the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S1920, the processing step S1930 may be omitted and the processing may proceed to step S1940.

[0232] In step S1940, once the image quality improving unit 404 has generated a group of high-quality images, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-quality images generated in step S1940 into an authenticity evaluation engine and performs an authenticity evaluation based on the output of the authenticity evaluation engine. Specifically, if the authenticity evaluation engine outputs an authentic label (genuine), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have been sufficiently improved in image quality. On the other hand, if the authenticity evaluation engine outputs a counterfeit label (fake), the authenticity evaluation unit 1807 evaluates that the generated high-quality images have not been sufficiently improved in image quality.

[0233] In step S1960, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved, the output unit 405 causes the high-quality image to be displayed on the display unit 20. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved, the output unit 405 causes the display unit 20 to display the input image.

[0234] As described above, the image processing device 1800 according to this embodiment further includes an authenticity evaluation unit 1807 that evaluates the image quality of a high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of the image. The authenticity evaluation engine includes a machine learning engine that uses as training data images generated by an image quality improvement engine that has lower (worse) image quality processing accuracy than the image quality improvement engine of the image quality improvement unit 404. The output unit 405 of the image processing device 1800 outputs the high-quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

[0235] As a result, with the image processing device 1800 according to this embodiment, the examiner can efficiently check high-quality images that have been sufficiently improved in quality.

[0236] Furthermore, the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine may be trained in coordination with each other to improve the efficiency and accuracy of both engines.

[0237] In this embodiment, the image quality improvement unit 404 generates one high-quality image and the authenticity evaluation unit 1807 evaluates the generated high-quality image, but the evaluation by the authenticity evaluation unit 1807 is not limited to this. For example, as in the second embodiment, when the image quality improvement unit 404 generates multiple high-quality images using multiple image quality improvement engines, the authenticity evaluation unit 1807 may be configured to evaluate at least one of the generated multiple high-quality images. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the generated multiple high-quality images, or may evaluate only the image specified by the examiner among the multiple high-quality images.

[0238] Furthermore, the output unit 405 may display on the display unit 20 the judgment result of the authenticity evaluation unit 1807 as to whether the high-quality image has been sufficiently improved in quality, and output the high-quality image in accordance with the examiner's instructions.

[0239] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1800. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

[0240] <Tenth embodiment> Next, an image processing device according to a tenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into multiple two-dimensional images, inputs the images to an image quality improvement engine, and combines the output images from the image quality improvement engine to generate a three-dimensional high-quality image.

[0241] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0242] The acquisition unit 401 according to this embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images). Also, for example, the three-dimensional image is a three-dimensional CT volume image composed of a group of axial tomographic images.

[0243] The image quality improving unit 404 is equipped with an image quality improving engine, as in the first embodiment. Note that a group of pairs of input data and output data, which are training data for the image quality improving engine, are composed of a group of two-dimensional images. The image quality improving unit 404 divides the acquired three-dimensional image into multiple two-dimensional images and inputs each two-dimensional image to the image quality improving engine. This allows the image quality improving unit 404 to generate multiple two-dimensional high-quality images.

[0244] The output unit 405 combines the multiple two-dimensional high-quality images generated by the image quality improving unit 404 for each of the two-dimensional images of the three-dimensional image, and outputs a three-dimensional high-quality image.

[0245] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are similar to those steps in the first embodiment, and therefore description thereof will be omitted. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that, if the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

[0246] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make this determination based on the shooting conditions of the three-dimensional image, or may make this determination based on the shooting conditions for the multiple two-dimensional images that make up the three-dimensional image. In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 inputs each of the multiple divided two-dimensional images into the image quality improvement engine to generate multiple two-dimensional high-quality images. Based on the acquired three-dimensional image, the image quality improvement unit 404 combines the generated multiple two-dimensional high-quality images to generate a three-dimensional high-quality image.

[0247] In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. The high-quality three-dimensional image may be displayed in any manner.

[0248] As described above, the image quality improvement unit 404 according to this embodiment divides a three-dimensional input image into multiple two-dimensional images and inputs them to the image quality improvement engine. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the image quality improvement engine to generate a three-dimensional high-quality image.

[0249] As a result, the image quality improving unit 404 according to this embodiment can improve the image quality of a three-dimensional image using an image quality improving engine that has been trained using teacher data of two-dimensional images.

[0250] Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data of the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0251] <Eleventh embodiment> Next, an image processing device according to an eleventh embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, improves the image quality of the plurality of two-dimensional images in parallel using a plurality of image quality improvement engines, and combines output images from the image quality improvement engines to generate a three-dimensional high-quality image.

[0252] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the tenth embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the tenth embodiment. Note that the configuration of the image processing device according to this embodiment is the same as those of the image processing devices according to the first and tenth embodiments, and therefore the configuration shown in FIG. 4 will be indicated by the same reference numerals and description thereof will be omitted.

[0253] The image quality improvement unit 404 according to this embodiment is provided with a plurality of image quality improvement engines similar to those in the tenth embodiment. Note that the plurality of image quality improvement engines provided in the image quality improvement unit 404 may be implemented in two or more device groups via a circuit or a network so as to enable distributed processing, or may be implemented in a single device.

[0254] As in the tenth embodiment, the image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improving unit 404 uses a plurality of image quality improving engines to improve the image quality of the plurality of two-dimensional images in a shared manner (in parallel), and generates a plurality of two-dimensional high-quality images. The image quality improving unit 404 combines a plurality of two-dimensional high-quality images output from a plurality of image quality improving engines based on the three-dimensional image to be processed, and generates a three-dimensional high-quality image.

[0255] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S510 to S530 and S550 according to this embodiment are the same as those steps in the tenth embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

[0256] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make this determination based on the shooting conditions of the three-dimensional image, or may make this determination based on the shooting conditions of the multiple two-dimensional images that make up the three-dimensional image.

[0257] In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided two-dimensional images into a plurality of image quality improvement engines, which perform image quality improvement processing in parallel to generate a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

[0258] In step S550, the output unit 405 displays the generated high-quality three-dimensional image on the display unit 20. The high-quality three-dimensional image may be displayed in any manner.

[0259] As described above, the image quality improvement unit 404 according to this embodiment includes multiple image quality improvement engines. The image quality improvement unit 404 divides a three-dimensional input image into multiple two-dimensional images and generates multiple two-dimensional high-quality images using the multiple image quality improvement engines in parallel. The image quality improvement unit 404 generates a three-dimensional high-quality image by integrating the multiple two-dimensional high-quality images.

[0260] As a result, the image quality improving unit 404 according to this embodiment can improve the image quality of a 3D image using an image quality improving engine that has been trained using teacher data of a 2D image. Furthermore, compared to the tenth embodiment, the image quality of a 3D image can be improved more efficiently.

[0261] The training data for the multiple image quality improvement engines may be different depending on the processing target to be processed by each image quality improvement engine. For example, a first image quality improvement engine may learn using training data for a first imaging region, and a second image quality improvement engine may learn using training data for a second imaging region. In this case, each image quality improvement engine can improve the image quality of two-dimensional images with higher accuracy.

[0262] Furthermore, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Furthermore, as in the first embodiment, the output data of the training data from the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0263] <Twelfth embodiment> Next, an image processing apparatus according to a twelfth embodiment will be described with reference to Figures 5 and 20. In this embodiment, an acquisition unit 401 acquires an input image from an image management system 2000, not from an image capturing device.

[0264] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device 400 according to the first embodiment, so the same reference numerals are used for the components shown in FIG. 4 and the description thereof will be omitted.

[0265] 20 shows a schematic configuration of an image processing device 400 according to this embodiment. The image processing device 400 according to this embodiment is connected to an image management system 2000 and a display unit 20 via any circuit or network. The image management system 2000 is a device and system that receives and stores images captured by any imaging device or processed images. The image management system 2000 can also transmit images in response to requests from connected devices, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS).

[0266] The acquisition unit 401 according to this embodiment can acquire an input image from the image management system 2000 connected to the image processing device 400. Furthermore, the output unit 405 can output a high-quality image generated by the image quality improvement unit 404 to the image management system 2000.

[0267] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S520 to S540 according to this embodiment are the same as those steps in the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S520, the processing step S530 may be omitted and the processing may proceed to step S540.

[0268] In step S510, the acquisition unit 401 acquires, as an input image, an image stored in the image management system 2000 from the image management system 2000 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the image management system 2000. Such a request may be issued, for example, when the image management system 2000 stores an image, before transmitting the stored image to another device, or when displaying the stored image on the display unit 20. The request may also be issued, for example, when a user operates the image management system 2000 to request image quality improvement processing, or when a high-quality image is used for an image analysis function provided in the image management system 2000.

[0269] The processing of steps S520 to S540 is the same as that in the first embodiment. Once the image quality improving unit 404 has generated a high-quality image in step S540, the processing proceeds to step S550. In step S550, if a high-quality image has been generated in step S540, the output unit 405 outputs the high-quality image to the image management system 2000 as an output image. If a high-quality image has not been generated in step S540, the output unit 405 outputs the input image to the image management system 2000 as an output image. Depending on the settings and implementation of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image management system 2000, or may convert the data format of the output image.

[0270] As described above, the acquisition unit 401 according to this embodiment acquires an input image from the image management system 2000. Therefore, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis based on an image stored in the image management system 2000, without increasing the invasiveness or labor of the photographer or the subject. The output high-quality image can be stored in the image management system 2000 or displayed on a user interface provided by the image management system 2000. The output high-quality image can be used for an image analysis function provided in the image management system 2000, or can be transmitted to another device connected to the image management system 2000 via the image management system 2000.

[0271] The image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via circuits or a network. Although these devices are separate devices in this embodiment, some or all of these devices may be integrated.

[0272] Furthermore, the output unit 405 may output the generated high-quality image to the image management system 2000 or other devices connected to the image processing device 400, as in the first embodiment.

[0273] <Thirteenth embodiment> Next, an image processing device according to a thirteenth embodiment will be described with reference to Figures 4, 5, 21A, and 21B. In this embodiment, an image quality improving unit uses a plurality of images as input images and generates one high-quality image.

[0274] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0275] The acquisition unit 401 according to this embodiment acquires a plurality of images as input data to be processed from the image capture device 10 or another device.

[0276] The image quality improving unit 404 according to this embodiment is equipped with an image quality improving engine similar to that of the first embodiment. The training data may also be similar to that of the first embodiment. The image quality improving unit 404 inputs each of the multiple images acquired by the acquisition unit 401 to the image quality improving engine, and performs a process of overlaying the multiple high-quality images output to generate a final high-quality image. Note that the image quality improving section 404 may align the positions of the multiple high-quality images using any method before overlaying the multiple high-quality images.

[0277] The output unit 405 displays the final high-quality image generated by the image quality improving unit 404 on the display unit 20. The output unit 405 may display a plurality of input images together with the final high-quality image on the display unit 20. The output unit 405 may also display the generated plurality of high-quality images together with the final high-quality image and the input image on the display unit 20.

[0278] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5 and Fig. 21A. Fig. 21A is a flow diagram of image quality improvement processing according to this embodiment. Note that the processing steps S510 to S530 according to this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0279] However, in step S510, the acquisition unit 401 acquires multiple images, and in steps S520 and S530, the shooting conditions are acquired for each of the multiple images, and it is determined whether the multiple images can be handled by the image quality improvement engine. If the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. Furthermore, if it is determined that some of the multiple images cannot be handled by the image quality improvement engine, the images may be excluded from subsequent processing.

[0280] In step S530, if the image quality improvement possibility determination unit 403 determines that the plurality of input images can be handled by the image quality improvement engine, the process proceeds to step S540. When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in Fig. 21A is started. In the image quality improvement process according to this embodiment, first, in step S2110, the image quality improvement unit 404 inputs each of the plurality of input images to the image quality improvement engine and generates a group of high-image-quality images.

[0281] Next, in step S2120, the image quality improvement unit 404 performs an overlay process on the generated group of high-quality images to generate a single final high-quality image. The overlay process may be performed by averaging such as arithmetic averaging or any other existing process. Furthermore, when overlaying, the image quality improvement unit 404 may align the positions of the multiple high-quality images using any method before overlaying them. Once the image quality improvement unit 404 has generated the final high-quality image, the process proceeds to step S550.

[0282] In step S550, the output unit 405 causes the display unit 20 to display the generated final high-quality image.

[0283] As described above, the image quality improvement unit 404 according to this embodiment generates a single final high-quality image from multiple input images. Because the image quality improvement by the image quality improvement engine is based on the input images, for example, if a lesion or other part is not properly displayed in a certain input image, the resulting high-quality image obtained by improving the image quality of that input image will have a low pixel value. On the other hand, in other input images capturing the same location, the lesion or other part may be properly displayed, resulting in a high pixel value in the resulting high-quality image obtained by improving the image quality of the other input image. Therefore, by overlaying these high-quality images, the low or high pixel value can be properly displayed, thereby generating a high-quality image with high contrast. Furthermore, by using a smaller number of input images than the number required for conventional overlaying, the tradeoffs, such as longer imaging times, as in the past, can be reduced.

[0284] This effect becomes more pronounced when using input images that use motion contrast data such as OCTA.

[0285] Since motion contrast data is obtained by detecting temporal changes in a subject over time intervals when the same location on the subject is repeatedly photographed, for example, only slight movement of the subject may be detected at a certain time interval. On the other hand, when photographing at a different time interval, the subject's movement may be detected as greater. Therefore, by overlaying high-quality motion contrast images from each case, it is possible to interpolate motion contrast that did not occur or was only slightly detected at a particular time. Therefore, this type of processing can generate a motion contrast image in which the contrast of more of the subject's movements is enhanced, allowing the examiner to more accurately grasp the subject's condition.

[0286] Therefore, when an image depicting an area that changes over time, such as an OCTA image, is used as the input image, a specific area of ​​the subject can be imaged in more detail by overlaying high-quality images acquired at different times.

[0287] In this embodiment, a single high-quality image is generated by generating a high-quality image from each of a plurality of input images and overlaying the high-quality images to generate a final single high-quality image, but the method of generating a single high-quality image from a plurality of input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in Figure 21B, when the image quality improvement process is started in step S540, image quality improvement unit 404 overlays the input images in step S2130 to generate a single overlaid input image.

[0288] Then, in step S2140, the image quality improvement unit 404 inputs one superimposed input image to the image quality improvement engine to generate one high-quality image. This type of image quality improvement process, like the above-described image quality improvement process, can appropriately display areas with low or high pixel values ​​in multiple input images, thereby generating a high-quality image with high contrast. This process also has a significant effect when a motion contrast image, such as the above-described OCTA image, is used as the input image.

[0289] When performing this image quality improvement process, the image quality improvement engine uses superimposed images of the same number of input images as the number of input images to be processed as training data for the image quality improvement engine, which allows the image quality improvement engine to perform appropriate image quality improvement processing.

[0290] Furthermore, in the image quality improvement process according to this embodiment and the other image quality improvement processes described above, the process of combining a group of high-quality images or a group of input images is not limited to overlay. For example, a single image may be generated by applying a MAP estimation process to these groups of images. Also, a single image may be generated by combining a group of high-quality images or a group of input images.

[0291] When generating a single image by combining a group of high-quality images or a group of input images, for example, an image having a wide range of gradations in the high-luminance region and an image having a wide range of gradations in the low-luminance region may be used as the input image. In this case, for example, an image obtained by improving the quality of an image having a wide range of gradations in the high-luminance region and an image obtained by improving the quality of an image having a wide range of gradations in the low-luminance region are combined. This makes it possible to generate an image that can express a wider range of brightness (dynamic range). In this case, the input data of the training data for the image quality improvement engine can be an image having a wide range of gradations in the high-luminance region or a low-quality image having a wide range of gradations in the low-luminance region, which are to be processed. Furthermore, the output data of the training data for the image quality improvement engine can be a high-quality image corresponding to the input data.

[0292] Alternatively, an image having a wide range of gradations in the high-brightness region and an image having a wide range of gradations in the low-brightness region may be synthesized, and the synthesized image may be enhanced in quality by an image enhancement engine. In this case, an image capable of expressing a wider range of brightness may be generated. In this case, the input data of the training data for the image enhancement engine may be an image obtained by synthesizing a low-quality image having a wide range of gradations in the high-brightness region and a low-quality image having a wide range of gradations in the low-brightness region, which are the processing targets. Furthermore, the output data of the training data for the image enhancement engine may be a high-quality image corresponding to the input data.

[0293] In these cases, a high-quality image engine can be used to improve the image quality of images that can express a wider range of brightness, making it possible to process using fewer images than before, and providing images suitable for image analysis at a lower cost.

[0294] As a method for capturing an image having a wide range of gradations in the high-luminance region and an image having a wide range of gradations in the low-luminance region, any method may be adopted, such as shortening or lengthening the exposure time of the image capturing device. Furthermore, the method for dividing the range of gradations is not limited to the low-luminance region and the high-luminance region, and any method may be used.

[0295] Furthermore, in the image quality improvement processing according to this embodiment, multiple image quality improvement engines may be used to process multiple input images in parallel. Note that, as in the first embodiment, the output unit 405 may output the generated high-quality image to the image capture device 10 or another device connected to the image processing device 400. Also, as in the first embodiment, the output data of the training data from the image quality improvement engine is not limited to high-quality images that have undergone overlay processing. In other words, high-quality images obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0296] <Fourteenth embodiment> Next, an image processing device according to a fourteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving unit uses a medium quality image generated from a plurality of low quality images as an input image and generates a high quality image.

[0297] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0298] The acquisition unit 401 according to this embodiment acquires a medium quality image obtained by superimposing a plurality of low quality images as input data to be processed from the image capture device 10 or another device. When superimposing the low quality images, any positioning process may be performed.

[0299] The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, the image quality improvement engine of this embodiment is designed to input a medium-quality image, which has a medium level of image quality, and output a high-quality image. A medium-quality image is a composite image generated by superimposing a group of low-quality images. A high-quality image is an image of higher quality than a medium-quality image. Furthermore, with regard to the group of pairs constituting the training data used for training the image quality improvement engine, the input data constituting each pair is a medium-quality image generated in the same manner as the medium-quality image, and the output data is a high-quality image.

[0300] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404 . In addition, the output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from multiple low-quality images.

[0301] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S520 to S550 according to this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0302] In step S510, the acquisition unit 401 acquires a medium-quality image as an input image from the image capturing device 10 or another device. Note that the acquisition unit 401 may acquire a medium-quality image generated by the image capturing device 10 as an input image in response to a request from the image capturing device 10. Such a request may be issued, for example, when the image capturing device 10 generates an image, before or after the image generated by the image capturing device 10 is saved in a storage device provided in the image capturing device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used in image analysis processing, etc.

[0303] The subsequent processing is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0304] As described above, the acquisition unit 401 according to this embodiment acquires, as an input image, a medium-quality image that is an image generated using multiple images of a specific part of the subject. In this case, the input image is a clearer image, so the image quality enhancement engine can generate a high-quality image with higher accuracy. Note that the number of low-quality images used to generate the medium-quality image may be smaller than the number of images used to generate a conventional superimposed image.

[0305] It should be noted that the medium quality image is not limited to an image obtained by superimposing a plurality of low quality images, but may be, for example, an image obtained by applying a MAP estimation process to a plurality of low quality images, or an image obtained by combining a plurality of low quality images. When combining a plurality of low quality images, images with different gradations may be combined.

[0306] Furthermore, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400. Furthermore, as in the first embodiment, the output data of the training data from the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0307] <Fifteenth embodiment> Next, an image processing device according to a fifteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving unit performs image quality improvement according to the first embodiment etc. as well as image size increase (enlargement) of an input image.

[0308] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0309] The acquisition unit 401 according to this embodiment acquires a low-size image (low-size image) as an input image. Note that a low-size image is an image that comprises fewer pixels than a high-size image (high-size image) output by a high-image-quality engine (described later). Specifically, for example, if the image size of the high-size image is 1024 pixels wide, 1024 pixels high, and 1024 pixels deep, the image size of the low-size image may be 512 pixels high, 512 pixels deep, and 512 pixels deep. In this regard, in this specification, "enhancing the image size" refers to a process of increasing the number of pixels per image and enlarging the image size.

[0310] The image quality improvement unit 404 according to this embodiment is provided with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine according to this embodiment is configured to reduce noise and enhance contrast of an input image, and to increase the image size of the input image. Therefore, the image quality improvement engine according to this embodiment is configured to input a low-size image and output a high-size image.

[0311] In this regard, for a group of pairs constituting the training data of the image quality improvement engine, the input data constituting each pair is a low-size image, and the output data is a high-size image. Note that the high-size images used for output data can be acquired from a device with higher performance than the imaging device that acquired the low-size images, or can be acquired by changing the settings of the imaging device. Also, if a group of high-size images already exists, the group of low-size images to be used as input data may be acquired by reducing the group of high-size images to the image size of images expected to be acquired from the imaging device 10. Also, as for the high-size images, low-size images superimposed on each other are used, as in the first embodiment, etc.

[0312] Note that the enlargement of the image size of the input image by the image quality improving unit 404 according to this embodiment is different from simple image enlargement, because the image size is obtained as training data from a device with higher performance than the image capture device 10 or by changing the settings of the image capture device 10. Specifically, the process of enlarging the image size of the input image by the image quality improving unit 404 according to this embodiment can reduce degradation in resolution compared to when the image is simply enlarged.

[0313] With this configuration, the image quality improving unit 404 according to this embodiment can reduce noise and enhance contrast from an input image and generate a high-quality image with a large image size.

[0314] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described here. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, the processing step S530 may be omitted after the processing step S520, and the processing may proceed to step S540.

[0315] In step S510, the acquisition unit 401 acquires a low-size image from the image capturing device 10 or another device as input data to be processed. Note that the acquisition unit 401 may acquire a low-size image generated by the image capturing device 10 as an input image in response to a request from the image capturing device 10. Such a request may be issued, for example, when the image capturing device 10 generates an image, before or after the image generated by the image capturing device 10 is saved in a storage device provided in the image capturing device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0316] The processing in steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image that has undergone noise reduction and contrast enhancement as a high-quality image and has a large image size. The processing thereafter is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0317] As described above, the image quality improving unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been enlarged. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing invasiveness or effort on the photographer or the subject.

[0318] In this embodiment, a high-quality image is generated by performing the high-quality image processing and the high-resolution processing according to the first embodiment etc. using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs the image quality processing according to the first embodiment etc. and another image quality improvement engine that performs the image size processing.

[0319] In this case, the image quality improvement engine that performs the image quality improvement process according to the first embodiment etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment etc. Furthermore, as input data of the training data for the image quality improvement engine that performs the image resizing process, the high-quality image generated by the image quality improvement engine according to the first embodiment etc. is used. Furthermore, as output data of the training data for the image quality improvement engine, the high-quality image generated by the image quality improvement engine according to the first embodiment etc. for an image acquired by a high-performance imaging device is used. In this way, the image quality improvement engine that performs the image resizing process can generate a final high-quality image by increasing the image size of the high-quality image that has been subjected to the image quality improvement process according to the first embodiment etc.

[0320] Furthermore, the image resizing process by the image quality improvement engine may be performed before the image quality improvement process by the image quality improvement engine according to the first embodiment, etc. In this case, the training data for the image quality improvement engine that performs the image resizing process is configured from a group of pairs of input data that is a low-size image acquired by the imaging device and output data that is a high-size image. Also, the training data for the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. is configured from a group of pairs of input data that is a high-size image and output data that is an image in which the high-size image is superimposed.

[0321] Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image and the image size has been enlarged.

[0322] In this embodiment, the image quality improvement process according to the first embodiment and the like has been described with reference to a configuration in which a superimposed image is used as output data of training data, but the output data is not limited to this, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0323] The output unit 405 may output the generated high-quality image to the photographing device 10 or another device connected to the image processing device 400, as in the first embodiment.

[0324] <16th embodiment> Next, an image processing device according to a sixteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving unit improves spatial resolution in addition to the image quality improvement according to the first embodiment and the like.

[0325] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0326] The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. The low spatial resolution image is an image with a lower spatial resolution than the high spatial resolution image output by the image quality improvement unit 404.

[0327] The image quality improvement unit 404 is provided with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast of an input image, and to increase the spatial resolution of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low spatial resolution image and output a high spatial resolution image.

[0328] In this regard, for the group of pairs that constitute the training data for the image quality improvement engine, the input data that constitutes each pair is a low spatial resolution image, and the output data is a high spatial resolution image. Note that the high spatial resolution image can be obtained from a device with higher performance than the imaging device that acquired the low spatial resolution image, or by changing the settings of the imaging device. Furthermore, as with the first embodiment, a high spatial resolution image is used in which a low spatial resolution image is superimposed.

[0329] With this configuration, the image quality improving unit 404 according to this embodiment can reduce noise and enhance contrast from an input image, and generate a high-quality image with high spatial resolution.

[0330] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S520, S530, and S550 according to this embodiment are similar to those steps in the first embodiment, and therefore will not be described here. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, the processing step S530 may be omitted after the processing step S520, and the processing may proceed to step S540.

[0331] In step S510, the acquisition unit 401 acquires a low spatial resolution image from the image capturing device 10 or another device as input data to be processed. Note that the acquisition unit 401 may acquire a low spatial resolution image generated by the image capturing device 10 as an input image in response to a request from the image capturing device 10. Such a request may be issued, for example, when the image capturing device 10 generates an image, before or after the image generated by the image capturing device 10 is saved in a storage device provided in the image capturing device 10, when the saved image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0332] The processing in steps S520 and S530 is the same as that in the first embodiment, and therefore a description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image that has undergone noise reduction and contrast enhancement as a high-quality image and has a high spatial resolution. The processing thereafter is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0333] As described above, the image quality improving unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image and the spatial resolution has been improved. As a result, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis without increasing invasiveness or effort on the photographer or the subject.

[0334] In this embodiment, a high-quality image is generated by performing the high-quality image processing and the high-resolution processing according to the first embodiment etc. using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include a high-quality image engine that performs the high-quality image processing according to the first embodiment etc. and another high-quality image engine that performs the high-resolution processing.

[0335] In this case, the image quality improvement engine that performs the image quality improvement processing according to the first embodiment etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment etc. Furthermore, as input data of the training data for the image quality improvement engine that performs the resolution improvement processing, a high-quality image generated by the image quality improvement engine according to the first embodiment etc. Furthermore, as output data of the training data for the image quality improvement engine, a high-quality image generated by the image quality improvement engine according to the first embodiment etc. from an image acquired by a high-performance imaging device is used. As a result, the image quality improvement engine that performs high spatial resolution processing can generate a final high-quality image with high spatial resolution from a high-quality image that has been subjected to image quality improvement processing according to the first embodiment, etc.

[0336] Furthermore, the high spatial resolution processing by the image quality improvement engine may be performed before the high spatial resolution processing by the image quality improvement engine according to the first embodiment, etc. In this case, the training data for the image quality improvement engine that performs the high spatial resolution processing is configured from a group of pairs of input data that is a low spatial resolution image acquired by an imaging device and output data that is a high spatial resolution image. Furthermore, the training data for the image quality improvement engine that performs image quality improvement processing according to the first embodiment, etc., is composed of a group of pairs of input data of a high spatial resolution image and output data of an image in which the high spatial resolution image is superimposed.

[0337] Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the spatial resolution has been improved.

[0338] In this embodiment, the image quality improvement process according to the first embodiment and the like has been described with reference to a configuration in which a superimposed image is used as output data of training data, but the output data is not limited to this, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0339] Furthermore, the image quality improving unit 404 may use an image quality improving engine to perform the image quality improving process according to the fifteenth embodiment in addition to the high spatial resolution process. In this case, at least one of noise reduction and contrast enhancement is performed compared to the input image, and an image with a larger image size and higher spatial resolution compared to the input image can be generated as a high-quality image. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing invasiveness or effort on the photographer or subject.

[0340] The output unit 405 may output the generated high-quality image to the photographing device 10 or another device connected to the image processing device 400, as in the first embodiment.

[0341] <Seventeenth embodiment> Next, an image processing apparatus according to a seventeenth embodiment will be described with reference to Figures 22 and 23. In this embodiment, an analysis unit performs image analysis on the high-quality image generated by the image quality improvement unit.

[0342] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0343] 22 shows a schematic configuration of an image processing device 2200 according to this embodiment. The image processing device 2200 according to this embodiment is provided with an analysis unit 2208 in addition to an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405. Note that the image processing device 2200 may be configured with a plurality of devices each provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, and therefore the same reference numerals as those in FIG. 4 are used to indicate the configuration, and description thereof will be omitted.

[0344] The analysis unit 2208 applies a predetermined image analysis process to the high-quality image generated by the image quality improvement unit 404. In the field of ophthalmology, the image analysis process includes any existing image analysis process, such as retinal layer segmentation, layer thickness measurement, optic disc three-dimensional shape analysis, lamina cribrosa analysis, vascular density measurement of OCTA images, and corneal shape analysis, for images acquired by OCT. Furthermore, the image analysis process is not limited to analysis processes in the field of ophthalmology, and also includes any existing analysis process in the field of radiology, such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

[0345] The output unit 405 can display the high-quality image generated by the image quality improving unit 404 on the display unit 20, as well as the analysis results of the image analysis processing performed by the analysis unit 2208. The output unit 405 can display only the image analysis results performed by the analysis unit 2208 on the display unit 20, or can output the image analysis results to the imaging device 10, an image management system, or other devices. The display form of the analysis results can be any form depending on the image analysis processing performed by the analysis unit 2208, and can be displayed, for example, as an image, numerical values, or text. The display form of the analysis results can also be an analysis result obtained by analyzing the high-quality image and superimposed on the high-quality image with any transparency. In other words, the display form of the analysis results can be an image (e.g., a two-dimensional map) obtained by blending the analysis result obtained by analyzing the high-quality image with the high-quality image with any transparency.

[0346] A series of image processing steps according to this embodiment will be described below with reference to Fig. 23, using an OCTA En-Face image as an example. Fig. 23 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S2310 to S2340 according to this embodiment are similar to the processing steps S510 to S540 in the first embodiment, and therefore will not be described here. Note that if the image quality of an input image is to be improved unconditionally regardless of the shooting conditions, the processing step S2330 may be omitted after the processing step S2320, and the processing may proceed to step S2340.

[0347] In step S2340, the image quality improvement unit 404 improves the image quality of the OCTA En-Face image, and the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high-quality image generated in step S2340. Image analysis of the high-quality OCTA En-Face image can be performed by applying any binarization process to detect blood vessel-equivalent areas (vascular regions) from the image. Area density can be analyzed by calculating the proportion of the image occupied by the detected blood vessel-equivalent areas. Furthermore, by thinning the binarized blood vessel-equivalent areas to an image with a line width of one pixel, the proportion of blood vessels that do not depend on their thickness (also known as skeleton density) can be calculated. These images can also be used to analyze the area and shape (e.g., circularity) of the avascular zone (FAZ). The analysis method can be to calculate the above-mentioned values ​​from the entire image, or to calculate values ​​for a specified region of interest (ROI) based on instructions from the examiner (user) using a user interface (not shown). The ROI setting does not necessarily have to be specified by the examiner; a predetermined region may be automatically specified. The various parameters described above are examples of analysis results related to blood vessels, and any parameters related to blood vessels may be used. The analysis unit 2208 may perform multiple image analysis processes. While an example of analysis of an En-Face image from OCTA has been shown here, the analysis unit 2208 may also perform retinal layer segmentation, layer thickness measurement, 3D optic disc shape analysis, lamina cribrosa analysis, and other similar processes on images simultaneously acquired by OCT. In this regard, the analysis unit 2208 may perform some or all of the multiple image analysis processes in response to instructions from the examiner via an input device.

[0348] In step S2360, the output unit 405 displays the high-quality image generated by the image quality improving unit 404 and the analysis results by the analysis unit 2208 on the display unit 20. The output unit 405 may output the high-quality image and the analysis results to separate display units or devices. Alternatively, the output unit 405 may display only the analysis results on the display unit 20. Furthermore, when the analysis unit 2208 outputs multiple analysis results, the output unit 405 may output some or all of the multiple analysis results to the display unit 20 or another device. For example, the analysis results related to blood vessels in the OCTA En-Face image may be displayed as a two-dimensional map on the display unit 20. Alternatively, values ​​indicating the analysis results related to blood vessels in the OCTA En-Face image may be superimposed on the OCTA En-Face image and displayed on the display unit 20.

[0349] As described above, the image processing device 2200 according to this embodiment further includes the analysis unit 2208 that performs image analysis on the high-quality image, and the output unit 405 displays the analysis results by the analysis unit 2208 on the display unit 20. In this way, the image processing device 2200 according to this embodiment uses high-quality images for image analysis, thereby improving the accuracy of the analysis.

[0350] Furthermore, as in the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 2200. Furthermore, as in the first embodiment, the output data of the training data from the image quality improvement engine is not limited to high-quality images that have been subjected to overlay processing. That is, it is also possible to use a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0351] <Eighteenth embodiment> Next, an image processing device according to an eighteenth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which an image quality improving unit generates a high-quality image by adding noise to an image during learning and learning the noise component.

[0352] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0353] The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or another device. An example of the configuration of a CNN in the image quality improvement unit according to this embodiment will be described with reference to FIG. 24. FIG. 24 shows an example of a machine learning model configuration in the image quality improvement unit 404. The configuration shown in FIG. 24 is composed of multiple layer groups that process and output a group of input values. As shown in FIG. 24, the types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merging layer. The convolution layer is a layer that performs convolution processing on a group of input values ​​according to parameters such as the set filter kernel size, the number of filters, the stride value, and the dilation value. The number of dimensions of the filter kernel size may also be changed depending on the number of dimensions of the input image. The downsampling layer is a process that reduces the number of output value groups to be smaller than the number of input value groups by thinning or merging the input value groups. Specifically, for example, Max Pooling processing is used. An upsampling layer is a process that increases the number of output values ​​by duplicating input values ​​or adding values ​​interpolated from the input values. Specifically, for example, linear interpolation processing is used. A synthesis layer is a layer that inputs values, such as the output values ​​of a layer or pixel values ​​constituting an image, from multiple sources and combines them by concatenating or adding them. In this configuration, the synthesis layer combines the pixel values ​​constituting the input image Im2410 with the values ​​output from the convolution processing block. The combined pixel values ​​are then converted into a high-quality image Im2420 in the final convolution layer. Although not shown, modifications to the CNN configuration may include incorporating a batch normalization layer or an activation layer using a rectifier linear unit after the convolution layer.

[0354] The image quality improvement engine of this embodiment inputs a low-quality image obtained by adding a first noise component to an image obtained from the image capture device 10 or another device, and performs training using an image obtained by adding a second noise component to an image obtained from the image capture device 10 or another device as output data, as a high-quality image. That is, in this embodiment, the same image is used as the teacher image during learning for both the low-quality image and the high-quality image, and the noise components in each image are different. Because the same image is used, there is no need to align the images when pairing them.

[0355] The noise components to be added include Gaussian noise, modeled noise specific to the target image, etc. However, the first and second noises are different noises. "Different noise" refers to noise added at different spatial locations (pixel positions) or with different noise values. For example, in the case of OCT, noise specific to the target image can be estimated based on data captured without a model eye or a test eye, and used as a noise model. In the case of OCTA, noise appearing in the avascular zone (FAZ) or in images captured with a model eye that schematically reproduces blood flow can be used as a noise model.

[0356] In the case of Gaussian noise, a standard deviation or variance is defined as the magnitude of the noise, and noise is randomly added to an image based on these values. The overall average value may remain unchanged as a result of adding random noise. That is, the average value of the noise added to each pixel of an image is set to 0. Here, the average does not need to be 0; it is sufficient to add noise of different patterns to the input data and the output data. It is also not necessary to add noise to both the input data and the output data; noise may be added to either one of them. Here, when no noise is added, for example, artifacts of blood vessels may appear in the image after image quality enhancement. This can be considered to occur when the difference between the images before and after image quality enhancement is relatively large. Therefore, the difference between the images before and after image quality enhancement may be reduced. In this case, during learning, two images obtained by adding different patterns of noise to a low-quality image and a high-quality image may be used as a pair of images, or two images obtained by adding different patterns of noise to a high-quality image may be used as a pair of images.

[0357] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404 . The output unit 405 may display the input image on the display unit 20 together with the high-quality image.

[0358] The subsequent processing is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0359] In this embodiment, a high-quality image is generated using an image obtained by adding a first noise component and a second noise component different from the first noise component to a low-quality image obtained from the image capture device 10 or another device, but the configuration for performing these processes is not limited to this. For example, the image to which noise is added may be a high-quality image that has been subjected to the overlay processing shown in the first embodiment, with the first and second noise components added. That is, a configuration may be used in which an image obtained by adding the first noise component to the overlay processing image is learned as a low-quality image, and an image obtained by adding the second noise component to the overlay processing image is learned as a high-quality image.

[0360] Furthermore, in this embodiment, an example of learning using the first and second noise components has been described, but this is not limiting. For example, learning may be performed by adding the first noise component only to the image to be determined as a low-quality image and not adding the noise component to the image to be determined as a high-quality image. The image in this case may be an image obtained from the imaging device 10 or another device, or an image obtained by superimposing the images may be used.

[0361] Furthermore, the magnitude of the noise component may be dynamically changed depending on the type of input image or for each rectangular region image to be learned. Specifically, adding a large noise value increases the effect of noise removal, while adding a small noise value decreases the effect of noise removal. Therefore, learning may be performed by adjusting the noise to be added depending on the conditions and type of the entire image or rectangular region image, for example, by reducing the value of the noise component to be added for a dark image and increasing the value of the noise component to be added for a bright image.

[0362] In this embodiment, the image capturing conditions are not specified, but learning is performed using images with various imaging ranges and different numbers of scans, front images of different imaging regions and different depths, etc.

[0363] The above describes images obtained from the image capture device 10 or other devices, noise images obtained by adding noise to those images, superimposition processed images, and images obtained by adding noise to superimposition processed images. However, these combinations are not limited to those described above, and low-quality images and high-quality images may be combined in any way.

[0364] <19th embodiment> Next, an image processing device according to a 19th embodiment will be described with reference to Figures 25 and 26. In this embodiment, an image quality improvement unit includes multiple image quality improvement engines, and generates multiple high-quality images for an input image. An example will be described in which a synthesis unit 2505 synthesizes the multiple high-quality images output from the multiple image quality improvement engines.

[0365] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0366] The acquisition unit 401 according to this embodiment acquires images as input data to be processed from the image capture device 10 or other devices.

[0367] The image quality improving unit 404 according to this embodiment is provided with a plurality of image quality improving engines, as in the second embodiment. Here, each of the plurality of image quality improving engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, noise components, and image resolution. The image quality improving unit 404 generates a high-quality image using a plurality of image quality improving engines corresponding to at least one of the imaging region, imaging angle of view, front images at different depths, noise components, and image resolution of the input image.

[0368] 26 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps S2610 and S2620 according to this embodiment are similar to the processing steps S510 and S520 according to the first embodiment, and therefore description thereof will be omitted. Note that if the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing step S2620, the processing step S2630 may be omitted and the processing may proceed to step S2640.

[0369] In step S2620, as in the first embodiment, once the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S2630. In step S2630, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image, as in the second embodiment.

[0370] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S2660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S2640. Note that, depending on the settings and implementation form of the image processing device 400, step S2640 may be performed even if it is determined that some of the shooting conditions cannot be handled by the image quality improvement engines, as in the first embodiment.

[0371] In step S2640, the image quality improving unit 404 inputs the input image acquired in step S2610 to each of the image quality improving engines, and generates a group of high-quality images.

[0372] In step S2650, the composition unit 2405 combines some high-quality images from the group of high-quality images generated in step S2640. Specifically, for example, the results of two high-quality images are combined: a first image quality improvement engine that has trained using paired images of a low-quality image acquired from the image capture device 10 and a high-quality image obtained by performing an averaging process or the like on a group of images acquired by capturing low-quality images multiple times as shown in the first embodiment; and a second image quality improvement engine that has trained using paired images in which noise has been added as shown in the 18th embodiment. As a composition method, averaging, weighted averaging, or the like can be used.

[0373] In step S2660, the output unit 405 displays the image synthesized in step S2650 on the display unit 20 or outputs it to another device. However, if it is determined in step S2630 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if the examiner specifies an input image or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0374] <Twentieth embodiment> Next, an image processing device according to a twentieth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which an image quality improvement unit uses the output result of a first image quality improvement engine to cause a second image quality improvement engine to generate a high-quality image.

[0375] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0376] The acquisition unit 401 according to this embodiment acquires images as input data to be processed from the image capture device 10 or other devices.

[0377] The image quality improvement unit 404 according to this embodiment is provided with a plurality of image quality improvement engines similar to those in the first embodiment. The image quality improvement unit of this embodiment is provided with a first image quality improvement engine that has learned using low-quality images acquired as input data from the image capture device 10 or another device, and medium-quality images generated from the plurality of low-quality images as output data. It also is provided with a second image quality improvement engine that has learned using images output from the first image quality improvement engine and images of higher quality than the medium-quality images as output data. Note that the medium-quality images are the same as those in the fourteenth embodiment, and therefore will not be described here.

[0378] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404 . In addition, the output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from multiple low-quality images.

[0379] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps S510 to S530 according to this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0380] In step S540, the image quality improvement unit 404 uses an image quality improvement engine to improve the image quality of the input image and generate a high-quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to a first image quality improvement engine, which generates a first high-quality image with improved image quality. The first high-quality image is then input to a second image quality improvement engine, which generates a second high-quality image. The image quality improvement engine generates a high-quality image that appears to have been overlaid using the input image, based on a machine learning model that has been machine-learned using training data. Therefore, the image quality improvement engine can generate a high-quality image that has reduced noise and enhanced contrast compared to the input image.

[0381] The subsequent processing is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0382] In this embodiment, high-quality images are generated using a first image quality improvement engine that has learned a pair of low-quality images and a medium-quality image obtained from the image capture device 10 or another device, and a second image quality improvement engine that has learned a pair of first high-quality images and a high-quality image, but the configuration for performing these processes is not limited to this. For example, the image pair learned by the first image quality improvement engine may be the engine that learns noise described in the 18th embodiment, and the second image quality improvement engine may learn a pair of first high-quality images and a high-quality image. As an opposite configuration, the first image quality improvement engine may learn a pair of low-quality images and a medium-quality image, and the second image quality improvement engine may learn an image to which noise has been added to the first high-quality image.

[0383] Furthermore, both the first image quality improvement engine and the second image quality improvement engine may be engines that learn noise as described in the eighteenth embodiment. In this case, for example, the first image quality improvement engine learns a pair of images in which first and second noises have been added to a high-quality image generated by superimposing processed images, and the second image quality improvement engine learns a pair of images in which first and second noises have been added to the first high-quality image generated by the first image quality improvement engine. Note that, although two image quality improvement engines have been described in this embodiment, this is not limiting, and a configuration in which a third and a fourth image quality improvement engine are further connected and processing may be used. By cleaning up the images used for learning, a network that is more likely to generate smoother and sharper images is constructed.

[0384] <Twenty-first embodiment> Next, an image processing apparatus according to a 21st embodiment will be described with reference to Figures 4 and 27. In the first embodiment, the image quality improving unit 404 includes one image quality improving engine. In contrast to this, in this embodiment, the image quality improvement unit includes a plurality of image quality improvement engines that perform machine learning using different training data, and generates a plurality of high-quality images from an input image.

[0385] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the second embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing devices according to the first and second embodiments. Note that, since the configuration of the image processing device according to this embodiment is the same as that of the image processing device according to the first and second embodiments, the configuration shown in FIG. 4 will be indicated by the same reference numerals, and description thereof will be omitted.

[0386] The image quality improvement unit 404 according to this embodiment is equipped with two or more image quality improvement engines, each of which has undergone machine learning using different training data. Here, a method for creating training data groups according to this embodiment will be described. First, pairs of original images as input data and superimposed images as output data, captured with various imaging ranges and different numbers of scans, are prepared. Taking OCT and OCTA as an example, for example, a first pair of images captured in a 3×3 mm area with 300 A-scans and 300 B-scans, and a second pair of images captured in a 10×10 mm area with 500 A-scans and 500 B-scans, are prepared. The scan density of the first pair of images and the second pair of images differs by two times. Therefore, these image groups are grouped separately. If there is an image group captured in a 6×6 mm area with 600 A-scans and 600 B-scans, this image group is grouped together with the first image group. That is, image groups with the same or nearly the same scan density (with an error of about 10%) are grouped together.

[0387] Next, the pairs are grouped by scan density to create a training data group. For example, a group of training data is created such that first training data is composed of a group of pairs obtained by photographing at a first scanning density, and second training data is composed of a group of pairs obtained by photographing at a second scanning density.

[0388] Then, separate image quality improvement engines are used to perform machine learning using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

[0389] Since each of these image quality improvement engines uses different teacher data to train its corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, a first image quality improvement engine improves the image quality of an input image acquired at a first scanning density to a high degree, but improves the image quality of an image acquired at a second scanning density to a low degree. Similarly, a second image quality improvement engine improves the image quality of an input image acquired at a second scanning density to a high degree, but improves the image quality of an image acquired at the first scanning density to a low degree.

[0390] On the other hand, there may be cases where it is not possible to collect a sufficient number of images with various shooting ranges and scan densities as training data during learning. In such cases, for such image groups, an image quality improvement engine that has learned noise components is prepared, as shown in the 18th embodiment.

[0391] The image quality improvement engine, which has learned about noise components, is less affected by the scan density at the time of shooting, so it is applied when an image with a scan density that has not been learned is input.

[0392] Since each training data set is composed of pairs grouped by scan density, the image groups constituting the pair groups have similar image quality trends. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, as long as the corresponding scan density is used. Note that the imaging conditions for grouping pairs of training data are not limited to scan density, but may also include the imaging region, images at different depths in frontal images, or a combination of two or more of these.

[0393] A series of image processing steps according to this embodiment will be described below with reference to Fig. 27. Fig. 27 is a flow diagram of a series of image processing steps according to this embodiment. Note that the processing steps of steps S2710 and S2720 are the same as steps S510 and S520 according to the first embodiment, and therefore description thereof will be omitted.

[0394] When the shooting conditions of the input image are acquired in step S2720, the process proceeds to step S2730. In step S2730, the image quality improvement possibility determination unit 403 uses the group of shooting conditions acquired in step S2720 to determine whether any of the group of image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

[0395] If the image quality improvement possibility determination unit 403 determines that the shooting conditions are outside the range, the process proceeds to step S2770. On the other hand, if the image quality improvement possibility determination unit 403 determines that the shooting conditions are within the range, the process proceeds to step S2740.

[0396] In step S2740, the image quality improvement unit 404 selects an image quality improvement engine from the group of image quality improvement engines to perform image quality improvement processing based on the shooting conditions of the input image and the teacher data information of the group of image quality improvement engines acquired in step S2720. Specifically, for example, for the scan density among the group of shooting conditions acquired in step S2720, an image quality improvement engine that has teacher data information related to the scan density and provides a high level of image quality improvement is selected. In the above example, if the scan density is the first scan density, the image quality improvement unit 404 selects the first image quality improvement engine.

[0397] On the other hand, in step S2770, the image quality improving unit 404 selects an image quality improving engine that has learned the noise component.

[0398] In step S2750, image quality improvement section 404 generates a high-quality image by improving the image quality of the input image using the image quality improvement engine selected in steps S2740 and S2770. Thereafter, in step S2760, output section 405 outputs the high-quality image in step S2750 and displays it on display section 20. Note that when displaying the high-quality image on display section 20, output section 405 may also display that the high-quality image was generated using the image quality improvement engine selected by image quality improvement section 404.

[0399] As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines, each trained using different learning data. Here, each of the plurality of image quality improvement engines has trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution. Furthermore, for data for which sufficient correct answer data (output data) could not be collected, learning was performed using noise components. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine corresponding to at least one of these.

[0400] With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

[0401] <Twenty-second embodiment> Next, an image processing device according to a 22nd embodiment will be described with reference to Figures 30 to 32. In this embodiment, a wide-angle image generation unit generates a wide-angle image (panoramic image) using a plurality of high-quality images generated by an image quality improvement unit.

[0402] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0403] FIG. 31(a) is a flow diagram of a series of image processing steps according to this embodiment. In step S3110, the acquisition unit 401 acquires multiple images (at least two) as input data from the imaging device 10 or another device. The multiple images are images of different positions on the same subject (e.g., a subject's eye), where the images do not completely overlap the subject but are images of parts that overlap. Taking the case of imaging a subject's eye as an example, by changing the position of the fixation light during imaging and having the subject's eye gaze at the fixation light, images of different positions on the same subject's eye can be acquired. It is desirable to change the position of the fixation light during image capture so that at least 20% of the overlapping areas between adjacent images are the same. FIG. 32(a) shows an example of OCTA En-Face images captured by changing the position of the fixation light so that adjacent images partially overlap. FIG. 32(a) shows an example of five images captured at different positions with the fixation light changed. Although five images are shown as an example in FIG. 32, the number is not limited to five and may be two or more.

[0404] The processing of step S3120 according to this embodiment is similar to the processing of step S520 according to the first embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S3120, the processing of step S3130 may be omitted and the processing may proceed to step S3140.

[0405] In step S3120, as in the first embodiment, once the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S3130. In step S3130, the image quality improvement feasibility determination unit 403 uses the acquired group of shooting conditions to determine whether the image quality improvement engine included in the image quality improvement unit 404 can handle the input image, as in the first embodiment.

[0406] If the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle multiple input images, the process proceeds to step S3160. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine can handle multiple input images, the process proceeds to step S3140. Note that, depending on the settings and implementation form of the image processing device 400, step S3140 may be performed even if the image quality improvement engine determines that it cannot handle some of the shooting conditions, as in the first embodiment.

[0407] In step S3140, the image quality improving unit 404 processes the multiple input images acquired in step S3110 to generate multiple high-quality images.

[0408] In step S3150, the wide-angle image generation unit 3005 combines several high-quality images from the group of high-quality images generated in step S3140. Specifically, an OCTA En-Face image will be used as an example. The multiple images do not completely overlap, but the OCTA En-Face images are captured so that adjacent images overlap in some areas. Therefore, the wide-angle image generation unit 3005 detects overlapping areas from the multiple OCTA En-Face images and performs alignment using the overlapping areas. By transforming the OCTA En-Face images based on the alignment parameters and combining the images, it is possible to generate an OCTA En-Face image with a wider range than a single OCTA En-Face image. At this time, the multiple OCTA En-Face images used as input have already been enhanced in image quality in step S3140, so the wide-angle OCTA En-Face image output in step S3150 has already been enhanced in image quality. FIG. 32( b ) shows an example of a wide-angle OCTA En-Face image generated by the wide-angle image generating unit 3005 . FIG. 32(b) shows an example generated by aligning the five images shown in FIG. 32(a). FIG. 32(c) shows the positional correspondence between FIG. 32(a) and FIG. 32(b). As shown in FIG. 32(c), Im3210 is positioned at the center, with Im3220 to Im3250 positioned around it. Note that multiple OCTA En-Face images can be generated by setting different depth ranges from three-dimensional motion contrast data. Therefore, while FIG. 32 shows an example of a surface image with a wide field of view, this is not limiting. For example, alignment can be performed using the surface OCTA En-Face image (Im2910) shown in FIG. 29, and OCTA En-Face images for other depth ranges can be deformed using the parameters determined there. Alternatively, a color image can be used as the input image for alignment, and a composite color image can be generated by combining the OCTA En-Face image of the surface layer as the RG component of the RGB components and the OCTA En-Face image to be aligned as the B component. Then, alignment can be performed on the composite color OCTA En-Face image, which combines layers from multiple depth ranges into a single image. By extracting only the B component from the aligned color OCTA En-Face image, a wide-angle OCTA En-Face image can be obtained, in which the target OCTA En-Face image has been aligned. Note that the target for image quality improvement is not limited to 2D OCTA En-Face images; it can also be 3D OCT or 3D motion contrast data itself. In this case, alignment can be performed on 3D data to generate wide-area 3D data. High-quality wide-angle images can be generated by extracting any cross-section (any plane in the X, Y, or Z directions) or any depth range (range in the Z direction) from wide-area 3D data.

[0409] In step S3160, the output unit 405 displays the image synthesized from the multiple images in step S3150 on the display unit 20 or outputs it to another device. However, if it is determined in step S3130 that the input image cannot be processed, the output unit 405 outputs the input image as the output image. Note that if the examiner specifies an input image or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0410] In this embodiment, high-quality images are generated from multiple input images, and the high-quality images are aligned to generate a single final high-quality wide-angle image. However, the method for generating a single high-quality image from multiple input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in Figure 31(b), a single wide-angle image may be generated first, and then the image quality improvement process may be performed on the wide-angle image to generate a single final high-quality wide-angle image.

[0411] This processing will be explained using FIG. 31(b), but the explanation of the processing that is the same as that in FIG. 31(a) will be omitted.

[0412] In step S3121, the wide-angle image generation unit 3005 combines the multiple images acquired in step S3110. The wide-angle image generation is the same as that described in step S3150, except that the input image is an image acquired from the image capture device 10 or another device, and is an image before being subjected to image quality enhancement.

[0413] In step S3151, the image quality improving unit 404 processes the high-quality image generated by the wide-angle image generating unit 3005 to generate a single high-quality wide-angle image.

[0414] With this configuration, the image processing device 400 according to this embodiment can generate a high-quality image with a wide angle of view.

[0415] With regard to the first to twenty-second embodiments, the display of a high-quality image on the display unit 20 by the output unit 405 is basically performed automatically in response to the generation of a high-quality image by the image quality improving unit 404 and the output of the analysis result by the analysis unit 2208. However, the display of a high-quality image may also be performed in response to an instruction from the examiner. For example, the output unit 405 may display on the display unit 20 an image selected in response to an instruction from the examiner from among the high-quality image generated by the image quality improving unit 404 and the input image. Furthermore, the output unit 405 may switch the display on the display unit 20 from the captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the output unit 405 may change the display of a low-quality image to the display of a high-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of a high-quality image to the display of a low-quality image in response to an instruction from the examiner. Furthermore, the image quality improvement unit 404 may start image quality improvement processing by the image quality improvement engine (input of an image to the image quality improvement engine) in response to an instruction from the examiner, and the output unit 405 may display the high-quality image generated by the image quality improvement unit 404 on the display unit 20. On the other hand, when an input image is captured by the imaging device 10, the image quality improvement engine may automatically generate a high-quality image based on the input image, and the output unit 405 may display the high-quality image on the display unit 20 in response to an instruction from the examiner. Note that these processes can also be performed on the output of the analysis results. That is, the output unit 405 may change the display of the analysis results of the low-quality image to the analysis results of the high-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the analysis results of the high-quality image to the analysis results of the low-quality image in response to an instruction from the examiner. Of course, the output unit 405 may change the display of the analysis results of the low-quality image to the display of the low-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of a low-quality image to a display of the analysis results of the low-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the analysis results of the high-quality image to a display of a high-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of a high-quality image to a display of the analysis results of the high-quality image in response to an instruction from the examiner.Furthermore, the output unit 405 may change the display of the analysis results of the low-quality image to a display of another type of analysis result of the low-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the analysis results of the high-quality image to a display of another type of analysis result of the high-quality image in response to an instruction from the examiner. Here, the display of the analysis results of the high-quality image may be a display in which the analysis results of the high-quality image are superimposed on the high-quality image with an arbitrary transparency. Furthermore, the display of the analysis results of the low-quality image may be a display in which the analysis results of the low-quality image are superimposed on the low-quality image with an arbitrary transparency. In this case, the change to the display of the analysis results may be, for example, a change to a state in which the analysis results are superimposed on the displayed image with an arbitrary transparency. Furthermore, the change to the display of the analysis results may be, for example, a change to a display of an image (e.g., a two-dimensional map) obtained by blending the analysis results and the image with an arbitrary transparency. Furthermore, the image processing device may be configured to start processing by a shooting location estimation engine, an image quality evaluation engine, an authenticity evaluation engine, and an evaluation unit in response to an instruction from the examiner. In the first to twenty-second embodiments, the output unit 405 may display high-quality images on the display unit 20 in any display mode. For example, the output unit 405 may display the input image and the high-quality image side by side, or may switch between them. The output unit 405 may also display the input image and the high-quality image in order according to the imaging region, imaging date and time, the facility where imaging was performed, etc. Similarly, the output unit 405 may display the image analysis results using the high-quality images in order according to any imaging conditions of the high-quality images and the input images corresponding to the high-quality images. Furthermore, the output unit 405 may display the image analysis results using the high-quality images in order for each analysis item.

[0416] <Twenty-third embodiment> Next, an image processing device according to a 23rd embodiment will be described with reference to Fig. 4, Fig. 29, and Fig. 33. In this embodiment, learning is performed using training data consisting of pairs of output data, which are high-quality images corresponding to input data. At this time, a single image quality improvement engine is generated using multiple high-quality output data generated by multiple image quality improvement engines.

[0417] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. The configuration of the image processing device according to this embodiment is similar to that of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0418] The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or another device. The generation of an image quality improvement engine in the image quality improvement unit 404 according to this embodiment will be described with reference to FIGS. 29 and 33. First, the first learning step according to this embodiment will be described with reference to FIG. 33(a). FIG. 33(a) shows an example of a group of pairs of input data and output data and a group of image quality improvement engines. Im3311 and Im3312 indicate a group of pairs of input data and output data. For example, this pair is the group of pairs in the surface layer (Im2910) shown in FIG. 29. 3313 indicates an image quality improvement engine trained using the group of pairs Im3311 and Im3312. The learning step in FIG. 33(a) may use a method using high-quality images generated by overlay processing as described in the first embodiment, or a method of learning noise components as described in the eighteenth embodiment. Alternatively, a combination of these may be used. Im3321 and Im3322 are pairs of input data and output data, for example, the pair group of the deep layer (Im2920) shown in FIG. 29. 3323 indicates an image quality improvement engine trained using the pair group of Im3321 and Im3322. Similarly, Im3331 and Im3332 are pairs of input data and output data, for example, the pair group of the outer layer (Im2930) shown in FIG. 29. 3333 indicates an image quality improvement engine trained using the pair group of Im3331 and Im3332. That is, in FIG. 33(a), training is performed for each image. Therefore, for example, in the case of the noise component described in the 18th embodiment, training can be performed using noise parameters appropriate for each image. In this case, the image quality improvement engine may include a machine learning engine obtained using training data to which noise corresponding to the state of at least a portion of a medical image is added to the at least a portion of the region. Here, the noise according to the state may be, for example, noise of a magnitude according to the pixel values ​​of at least a part of the region, or may be small noise when at least a part of the region has few characteristics (for example, small pixel values, low contrast, etc.).Furthermore, the noise according to the above state may be large noise, for example, when at least a portion of the area has many features (e.g., large pixel values, high contrast, etc.). The image quality improvement engine may include a machine learning engine obtained using training data including a plurality of front images to which different levels of noise have been added for at least two depth ranges among the plurality of depth ranges. In this case, for example, in a depth range corresponding to a front image with few features (e.g., small pixel values), the training data may be front images to which small noise has been added. For example, in a depth range corresponding to a front image with many features (e.g., large pixel values), the training data may be front images to which large noise has been added. Note that, for a depth range corresponding to a front image with a medium level of features, the training data may be front images to which medium-level noise has been added. Here, two adjacent depth ranges in the depth direction may partially overlap each other.

[0419] Next, image inference in this embodiment will be described using Figure 33(b). Figure 33(b) shows an image generated using image quality improvement engines 3313 to 3333 that have been trained in Figure 33(a). For example, when low-quality surface image Im3310 is input to image quality improvement engine 3313 that has been trained using multiple surface images, it outputs high-quality surface image Im3315. Similarly, when low-quality deep image Im3320 is input to image quality improvement engine 3323 that has been trained using multiple deep images, it outputs high-quality deep image Im3325. Similarly, when low-quality outer layer image Im3330 is input to image quality improvement engine 3333 that has been trained using multiple outer layer images, it outputs high-quality outer layer image Im3335.

[0420] Next, the second learning in this embodiment will be described using FIG. 33(c). FIG. 33(c) shows how one image quality improvement engine 3300 is trained using a plurality of image pair groups of different types. Im3310 indicates a low-quality surface image, Im3315 indicates a pair group of high-quality surface images, Im3320 indicates a low-quality deep image, Im3325 indicates a pair group of high-quality deep images, Im3330 indicates a low-quality outer image, and Im3335 indicates a pair group of high-quality outer images. That is, the image quality improvement engine 3300 is generated using training data consisting of a pair group of output data, which is a high-quality image generated using the image quality improvement engine trained in the first learning, and low-quality input data. As described above, the image quality improvement engine 3300 can generate high-quality images suitable for image diagnosis, with reduced noise and high contrast, from various types of input images.

[0421] The output unit 405 displays the high-quality image generated by the image quality improving unit 404 on the display unit 20. The output unit 405 may also display the input image on the display unit 20 together with the high-quality image.

[0422] The subsequent processing is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0423] In this embodiment, the OCTA En-Face images are described using three layers at different depths. However, the image types are not limited to this. It is also possible to increase the number of types by generating OCTA En-Face images with different depth ranges set by changing the reference layer and offset value. The image types are not limited to differences in depth, but may also be different for each region. For example, they may be different imaging locations, such as the anterior segment and the posterior segment. Furthermore, the images are not limited to OCTA En-Face images, but may also be luminance En-Face images generated from OCT data. Furthermore, the first learning may be performed separately, and then the OCTA En-Face images and luminance En-Face images may be learned together in the second learning. Furthermore, in addition to En-Face images, different imaging devices, such as tomographic images, SLO images, fundus photographs, and fluorescent fundus photographs, may also be used.

[0424] Although the example in which the second learning results in one image quality improvement engine has been described, the number of image quality improvement engines does not necessarily have to be one. It is sufficient for the image quality improvement engine to be configured to learn using pairs of output data of the image quality improvement engine generated in the first learning and low-image-quality input data. Furthermore, in the second learning, while FIG. 33(c) shows an example in which learning is performed simultaneously using multiple image pair groups of different types, this is not limiting and transfer learning may also be used. For example, after learning using a pair group of surface images Im3310 and Im3315, the network may be used to learn a pair group of deep images Im3320 and Im3325, ultimately generating the image quality improvement engine 3300.

[0425] With this configuration, the image quality improving unit 404 according to this embodiment can generate high-quality images more effectively for various types of images.

[0426] <Twenty-fourth embodiment> Next, an image processing device according to a 24th embodiment will be described with reference to Fig. 34. In this embodiment, an example will be described in which the output unit 405 displays the processing results of the image quality improvement unit 404 on the display unit 20. Note that although this embodiment will be described using Fig. 34, the display screen is not limited to this. The image quality improvement process can also be applied to a display screen that displays multiple images obtained at different dates and times side by side, such as in follow-up observation. The image quality improvement process can also be applied to a display screen, such as an imaging confirmation screen, on which the examiner checks whether imaging was successful immediately after imaging.

[0427] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0428] The output unit 405 can display a plurality of high-quality images generated by the image quality improvement unit 404 and low-quality images that have not been subjected to image quality improvement on the display unit 20. This makes it possible to output low-quality images and high-quality images in response to instructions from the examiner.

[0429] An example of the interface 3400 will be shown below with reference to FIG. 34. 3400 represents the entire screen, 3401 represents a patient tab, 3402 represents an imaging tab, 3403 represents a report tab, and 3404 represents a settings tab. The diagonal lines in the report tab 3403 indicate the active state of the report screen. In this embodiment, an example of displaying the report screen will be described. Im3405 represents an SLO image, and Im3406 represents an OCTA En-Face image shown in Im3407 superimposed on the SLO image Im3405. Here, the SLO image is a frontal image of the fundus acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown). Im3407 and Im3408 represent OCTA En-Face images, Im3409 represents a luminance En-Face image, and Im3411 and Im3412 represent tomographic images. Buttons 3413 and 3414 show the boundary lines of the upper and lower ranges of the OCTA En-Face images shown in Im3407 and Im3408, respectively, superimposed on the tomographic image. Button 3420 is a button for specifying the execution of image quality improvement processing. Of course, as will be described later, button 3420 may also be a button for instructing the display of a high-quality image.

[0430] In this embodiment, image quality improvement processing is performed by specifying button 3420, or whether or not to perform the processing is determined based on information saved (stored) in a database. First, an example will be described in which the display of a high-quality image and a low-quality image are switched by specifying button 3420 in response to an instruction from the examiner. Note that the image targeted for image quality improvement processing will be described as an OCTA En-Face image. When the examiner specifies report tab 3403 to transition to the report screen, low-quality OCTA En-Face images Im3407 and Im3408 are displayed. Thereafter, when the examiner specifies button 3420, the image quality improvement unit 404 performs image quality improvement processing on images Im3407 and Im3408 displayed on the screen. After the image quality improvement processing is completed, the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the report screen. Note that Im3406 is an image in which Im3407 is superimposed on SLO image Im3405, and thus Im3406 also displays an image that has been subjected to image quality improvement processing. Then, the display of button 3420 is changed to an active state, and a display is displayed that indicates that image quality improvement processing has been performed. Here, execution of processing by the image quality improvement unit 404 does not need to be limited to the timing when the examiner designates button 3420. Since the types of OCTA En-Face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, image quality improvement processing may be performed when transitioning to the report screen. Then, when button 3420 is pressed, the output unit 405 may display a high-quality image on the report screen. Furthermore, the types of images to be subjected to image quality improvement processing in response to an instruction from the examiner or when transitioning to the report screen do not need to be limited to two types. Processing may be performed on images that are likely to be displayed, for example, multiple OCTA En-Face images such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940) shown in Fig. 29. In this case, the images obtained after image quality improvement processing may be temporarily stored in memory or in a database.

[0431] Next, a case where image quality improvement processing is performed based on information saved (stored) in the database will be described. If the state for performing image quality improvement processing is saved in the database, when transitioning to the report screen is made, a high-quality image obtained by performing image quality improvement processing is displayed by default. Then, by displaying button 3420 in an active state by default, it is possible to configure the screen so that the examiner knows that a high-quality image obtained by performing image quality improvement processing is being displayed. If the examiner wants to display a low-quality image before image quality improvement processing, the examiner can display the low-quality image by specifying button 3420 to cancel the active state. If the examiner wants to return to a high-quality image, the examiner specifies button 3420. Whether or not to perform image quality improvement processing on the database may be specified hierarchically, such as for all data stored in the database, or for each piece of imaging data (each examination). For example, if the state of performing image quality improvement processing is saved for the entire database, and the examiner saves a state in which image quality improvement processing is not performed for individual imaging data (each examination), the next time that data is displayed, the image quality improvement processing will not be performed. A user interface (e.g., a save button) (not shown) may be used to save the state of image quality improvement processing for each piece of imaging data (each examination). Furthermore, when transitioning to other imaging data (other examination) or other patient data (e.g., changing to a display screen other than the report screen in response to an instruction from the examiner), the state of performing image quality improvement processing may be saved based on the display state (e.g., the state of button 3420). In this way, if the state of performing image quality improvement processing is not specified for each piece of imaging data (each examination), processing is performed based on the information specified for the entire database. If the state of performing image quality improvement processing is specified for each piece of imaging data (each examination), processing can be performed individually based on that information.

[0432] In this embodiment, an example is shown in which Im3407 and Im3408 are displayed as OCTA En-Face images, but the OCTA En-Face image to be displayed can be changed by the examiner. Therefore, the following describes how the image is changed when execution of image quality improvement processing is specified (button 3420 is active).

[0433] The image is changed using a user interface (e.g., a combo box) not shown. For example, when the examiner changes the image type from superficial to choroidal vascular network, the image quality improvement unit 404 performs image quality improvement processing on the choroidal vascular network image, and the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the report screen. That is, in response to an instruction from the examiner, the output unit 405 may change the display of a high-quality image in a first depth range to the display of a high-quality image in a second depth range that is at least partially different from the first depth range. At this time, the output unit 405 may change the display of a high-quality image in the first depth range to the display of a high-quality image in the second depth range by changing the first depth range to the second depth range in response to an instruction from the examiner. Note that, as described above, if a high-quality image has already been generated for an image that is likely to be displayed when the report screen transitions, the output unit 405 may display the generated high-quality image. The method for changing the image type is not limited to the above. It is also possible to generate OCTA En-Face images with different depth ranges set by changing the reference layer and offset value. In this case, when the reference layer or offset value is changed, the image quality improvement unit 404 performs image quality improvement processing on any OCTA En-Face image, and the output unit 405 displays the high-quality image on the report screen. The reference layer and offset value can be changed using a user interface (e.g., a combo box or text box) (not shown). The generation range of the OCTA En-Face image can be changed by dragging either the boundary lines 3413 and 3414 superimposed on the tomographic images Im3411 and Im3412 (moving the layer boundary). When the boundary line is changed by dragging, execution commands for the image quality improvement processing are executed continuously. Therefore, the image quality improvement unit 404 may always process the execution command, or may execute the command after the layer boundary is changed by dragging. Alternatively, the execution of the image quality improvement process may be commanded continuously, but when the next command arrives, the previous command may be cancelled and the latest command may be executed. Note that the image quality improvement process may take a relatively long time. For this reason, regardless of the timing at which the command is executed, it may take a relatively long time for a high-quality image to be displayed. Therefore, from the time a depth range for generating an OCTA En-Face image is set in response to an instruction from the examiner until a high-quality image is displayed, an OCTA En-Face image (low-quality image) corresponding to the set depth range may be displayed. That is, when the depth range is set, an OCTA En-Face image (low-quality image) corresponding to the set depth range may be displayed, and when the image quality improvement process is completed, the display of the OCTA En-Face image (low-quality image) may be changed to a high-quality image. Furthermore, from the time the depth range is set until a high-quality image is displayed, information indicating that the image quality improvement process is being performed may be displayed. Note that these may be applied not only when the execution of the image quality improvement process has already been specified (the button 3420 is active), but also, for example, when the execution of the image quality improvement process is instructed in response to an instruction from the examiner and until a high-quality image is displayed.

[0434] In this embodiment, an example has been shown in which different layers are displayed as OCTA En-Face images in Im3407 and Im3408, and low-quality and high-quality images are displayed by switching between them, but this is not limiting. For example, a low-quality OCTA En-Face image may be displayed in Im3407, and a high-quality OCTA En-Face image may be displayed side by side in Im3408. When images are displayed by switching between them, it is easy to compare areas that have changed because the images are switched in the same location. When images are displayed side by side, it is easy to compare the entire images because the images can be displayed simultaneously.

[0435] Next, the execution of image quality improvement processing during screen transitions will be described using FIGS. 34(a) and (b). FIG. 34(b) is an example of a screen in which the OCTA En-Face image Im3407 in FIG. 34(a) is enlarged. Similarly to FIG. 34(a), a button 3420 is displayed in FIG. 34(b). The screen transition from FIG. 34(a) to FIG. 34(b) is achieved by, for example, double-clicking the OCTA En-Face image Im3407, and the transition from FIG. 34(b) to FIG. 34(a) is achieved by clicking the close button 3430. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may also be used. If the execution of image quality improvement processing is specified during screen transitions (button 3420 is active), this state is maintained during screen transitions. That is, when a high-quality image is displayed on the screen of FIG. 34(a) and the screen of FIG. 34(b) is transitioned to the screen of FIG. 34(b), the high-quality image is also displayed on the screen of FIG. 34(b). Then, button 3420 is set to the active state. The same applies when transitioning from Fig. 34(b) to Fig. 34(a). In Fig. 34(b), it is also possible to switch the display to a low-quality image by specifying button 3420. Regarding screen transitions, the transition is not limited to the screens shown here, and as long as the transition is to a screen that displays the same captured data, such as a display screen for follow-up observation or a display screen for panoramic images, the transition is performed while maintaining the display state of the high-quality image. That is, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed on the display screen after the transition. For example, if the button 3420 on the display screen before the transition is in an active state, a high-quality image is displayed on the display screen after the transition. Also, for example, if the active state of the button 3420 on the display screen before the transition is released, a low-quality image is displayed on the display screen after the transition. Note that when the button 3420 on the follow-up observation display screen becomes active, multiple images obtained at different dates and times (different examination dates) displayed side by side on the follow-up observation display screen may be switched to high-quality images. That is, when button 3420 on the follow-up observation display screen becomes active, the result may be reflected collectively on multiple images acquired at different dates and times. An example of the follow-up observation display screen is shown in FIG. 38. When tab 3801 is selected in response to an instruction from the examiner, the follow-up observation display screen is displayed as shown in FIG. 38. At this time, the examiner can change the depth range of the En-Face image by selecting it from the preset depth range set (3802 and 3803) displayed in the list box. For example, the superficial retina is selected in list box 3802, and the deep retina is selected in list box 3803. The upper display area displays the analysis results of the En-Face image of the superficial retina, and the lower display area displays the analysis results of the En-Face image of the deep retina. That is, when a depth range is selected, the display is collectively changed to a side-by-side display of the analysis results of multiple En-Face images in the selected depth range for multiple images acquired at different dates and times. At this time, when the display of the analysis results is deselected, the display may be collectively changed to a side-by-side display of multiple En-Face images acquired at different dates and times. Then, when button 3420 is designated in response to an instruction from the examiner, the display of multiple En-Face images is collectively changed to the display of multiple high-quality images. Furthermore, if the display of analysis results is in a selected state, when button 3420 is designated in response to an instruction from the examiner, the display of analysis results of multiple En-Face images is collectively changed to the display of analysis results of multiple high-quality images. Here, the display of the analysis results may be superimposed on the image with an arbitrary transparency. In this case, the change to the display of the analysis results may be, for example, a change to a state in which the analysis results are superimposed on the displayed image with an arbitrary transparency. Furthermore, the change to the display of the analysis results may be, for example, a change to the display of an image (e.g., a two-dimensional map) obtained by blending the analysis results and the image with an arbitrary transparency. Furthermore, the type of layer boundary and the offset position used to specify the depth range can be collectively changed from user interfaces such as 3805 and 3806, respectively.The depth ranges of multiple En-Face images taken at different dates and times may be changed simultaneously by displaying a tomographic image and moving the layer boundary data superimposed on the tomographic image in response to an operator's instruction. In this case, multiple tomographic images taken at different dates and times may be displayed side by side, and when the above movement is made on one tomographic image, the layer boundary data may be moved similarly on the other tomographic images. The image projection method and the pr...

Claims

1. an acquisition unit that acquires a first image, which is a medical image of a predetermined part of a subject; a display control unit that controls the display unit to display either a second image acquired by inputting the first image into the trained model as an input image of the trained model, or the first image; A case where a display screen is changed from one of a first display screen on which either the first image or the second image is displayed to a second display screen on which either the first image or the second image is displayed to the other display screen in response to an instruction from an examiner, If the first image is displayed on one of the display screens, the change is made so that the first image is displayed on the other display screen; and When the second image is displayed on one of the display screens, the change is performed so that the second image is displayed on the other display screen.

2. A medical image processing device as described in claim 1, wherein a user interface for specifying whether to display the first image or the second image is displayed in accordance with instructions from an examiner, and when the first image and a user interface indicating the state in which the first image has been designated are displayed on one of the display screens, the change is made so that the first image and a user interface indicating the state in which the first image has been designated are displayed on the other display screen, and when the second image and a user interface indicating the state in which the second image has been designated are displayed on one of the display screens, the change is made so that the user interface indicating the state in which the second image and the second image have been designated are displayed on the other display screen.

3. 3. The medical image processing device according to claim 2, wherein the display control unit controls the display unit to display either the analysis results of the first image or the analysis results of the second image, and controls the display unit to change from one of the display of the analysis results of the first image or the display of the analysis results of the second image to the other display in accordance with an instruction from an examiner via the displayed user interface, and when the analysis results of the first image are displayed on one of the display screens, the change is made so that the analysis results of the first image are displayed on the other display screen, and when the analysis results of the second image are displayed on one of the display screens, the change is made so that the analysis results of the second image are displayed on the other display screen.

4. 4. The medical image processing device according to claim 2, wherein the display control unit controls the display unit so that a plurality of medical images corresponding to a plurality of different depth ranges of the specified portion in the three-dimensional medical image data of the specified portion are displayed side by side on the first display screen as one of the first image and the second image, controls the display unit so that the display of the plurality of medical images is simultaneously changed from one of the first image and the second image to the other in accordance with an instruction from an examiner via the displayed user interface, and controls the display unit so that, when one medical image of the plurality of medical images is selected in accordance with an instruction from the examiner, the display screen is changed from the first display screen to the second display screen on which the selected medical image is enlarged and displayed.

5. 5. The medical image processing device according to claim 1, wherein the second image displayed on the first display screen and the second display screen is a composite image obtained by combining pixel values ​​of corresponding pixels in the first image and the second image at a ratio obtained using information regarding at least a portion of an area in at least one of the first image and the second image, or at a ratio that can be changed in response to instructions from an examiner.

6. The medical image processing device according to claim 1 , wherein the trained model is a trained model obtained using training data to which noise is added to at least a partial region of a medical image.

7. The medical image processing device according to claim 1 , wherein the trained model is a trained model obtained using training data to which noise corresponding to the state of at least a portion of a region of a medical image is added to the at least a portion of the region.

8. 8. The medical image processing device according to claim 1, wherein the trained model is a trained model obtained using training data in which noise of a magnitude corresponding to pixel values ​​of at least a portion of a region of a medical image is added to the at least a portion of the region.

9. The medical image processing device according to claim 1 , wherein the trained model is a trained model obtained using training data including, as paired images, a plurality of medical images to which noise of different patterns has been added.

10. 10. The medical image processing apparatus according to claim 1, wherein the trained model is a trained model obtained using training data including, as paired images, a plurality of medical images obtained by adding noise of different patterns to medical images obtained by superposition processing.

11. The apparatus further includes a designation unit for designating a part of a depth range of the predetermined region in the three-dimensional medical image data of the predetermined region in response to an instruction from an examiner, the acquisition unit acquires, as the first image, a medical image corresponding to the specified partial depth range; The medical image processing device according to claim 1 , wherein the trained model is a trained model obtained using training data including a plurality of medical images corresponding to a plurality of depth ranges of a predetermined part of a subject.

12. The medical image processing device according to claim 11, wherein the trained model is a trained model obtained using training data in which noise of a magnitude corresponding to information regarding pixel values ​​of at least a portion of a region in each of at least two of the plurality of depth ranges is added to at least a portion of a region in each of a plurality of medical images corresponding to the plurality of depth ranges.

13. 13. The medical image processing apparatus according to claim 11, further comprising a generation unit that generates a wide-angle image using a plurality of second images obtained from a plurality of first images obtained by photographing different positions of the specified part in a direction intersecting a depth direction of the specified part so that partial regions of the plurality of adjacent medical images corresponding to the specified partial depth range overlap.

14. 14. The medical image processing device according to claim 1, wherein the second image is generated by dividing the first image into a plurality of two-dimensional images and inputting the divided images into the trained model, and integrating a plurality of output images from the trained model.

15. The trained model is a trained model obtained using training data including a plurality of medical images as paired images whose positional relationships correspond to each other, The medical image processing device according to claim 14 , wherein the first image is divided into the plurality of two-dimensional images with an image size corresponding to an image size of the paired images and input to the trained model.

16. 16. The medical image processing apparatus according to claim 14, wherein the trained model is a trained model obtained using training data including images of a plurality of partial regions that are set so that adjacent partial regions partially overlap with each other in an area including the medical image and its external periphery.

17. 17. The medical image processing device according to claim 1, wherein the trained model is a trained model obtained using training data including images obtained by OCTA imaging using an OCT imaging device with higher performance than the OCT imaging device used for OCTA imaging of the first image, or images obtained by an OCTA imaging process that requires more labor than the OCTA imaging process of the first image.

18. The medical image processing device according to claim 1 , wherein the trained model is a trained model obtained using training data including medical images obtained by superposition processing.

19. The medical image processing apparatus according to claim 1 , wherein the medical image is an OCTA front image or an OCT tomographic image.

20. 20. The medical image processing device according to claim 1, wherein the second image of a partial image of the three-dimensional medical image of the specified region for which a partial image has not yet been selected by an examiner's instruction is acquired before an instruction to select the partial image is input.

21. acquiring a first image, which is a medical image of a predetermined region of a subject; and controlling a display unit to display either a second image acquired by inputting the first image into the trained model as an input image of the trained model, or the first image; A case where a display screen is changed from one of a first display screen on which either the first image or the second image is displayed to a second display screen on which either the first image or the second image is displayed to the other display screen in response to an instruction from an examiner, If the first image is displayed on one of the display screens, the change is made so that the first image is displayed on the other display screen; and When the second image is being displayed on one of the display screens, the change is made so that the second image is displayed on the other display screen.

22. A program that, when executed by a processor, causes the processor to perform each step of the medical image processing method according to claim 21.

Images