Medical image processing device, medical image processing method and program

Abstract

To provide a medical image processing device that can generate images more suitable for image diagnosis than conventional ones.SOLUTION: A medical image processing device comprises: an acquisition unit for acquiring a first image which is a medical image of a prescribed portion of a subject; a high-quality image enhancement unit for generating, from the first image, a second image whose image quality is enhanced compared to the first image, using a high-quality image enhancement engine including a machine learning engine; and a display control unit for causing a display unit to display a composite image obtained by combining the first image and the second image based on a ratio obtained by using information regarding at least some regions in the first image.SELECTED DRAWING: Figure 34

Description

[Technical field]

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art In the medical field, images are acquired by various imaging devices and image diagnosis is performed by medical professionals in order to identify a disease in a subject or observe the severity of the disease. In the field of radiology, the types of imaging devices include X-ray imaging devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices, etc. In the field of ophthalmology, the types of imaging devices include fundus cameras, scanning laser ophthalmoscopes (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

[0003] In order to perform image diagnosis accurately and complete it in a short time, it is important that the images acquired by the imaging device have high image quality, such as low noise, high resolution and spatial resolution, and appropriate gradation. In addition, images in which the area or lesion to be observed is highlighted can also be useful.

[0004] However, in many imaging devices, some trade-off is required to obtain images suitable for image diagnosis, such as high image quality. For example, there is a method of purchasing a high-performance imaging device to obtain high-image quality images, but this often requires a larger investment than a low-performance imaging device.

[0005] In addition, for example, in CT, the subject's radiation dose may need to be increased to obtain an image with less noise. In addition, for example, in MRI, a contrast agent with a risk of side effects may be used to obtain an image that highlights the area to be observed. In addition, for example, in OCT, if the area to be photographed is large or high spatial resolution is required, the imaging time may be longer. In addition, for example, some imaging devices require multiple images to be taken to obtain high-quality images, which takes time to capture.

[0006] Patent Document 1 discloses a technique for converting previously acquired images into images with higher resolution using an artificial intelligence engine in order to respond to rapid advances in medical technology and simple imaging in emergencies. With this technique, for example, an image acquired by simple imaging with little cost can be converted into an image with higher resolution. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2018-5841 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, even a high-resolution image may not be suitable for image diagnosis. For example, even if the image has high resolution, if there is a lot of noise or low contrast, it may not be possible to properly grasp the object to be observed.

[0009] In response to this, one object of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program that can generate images that are more suitable for image diagnosis than ever before. [Means for solving the problem]

[0010] A medical image processing device according to one embodiment of the present invention includes an acquisition unit that acquires a first image, which is a medical image of a specific part of a subject, a quality improvement unit that uses a quality improvement engine including a machine learning engine to generate a second image from the first image, the second image having higher quality than the first image, and a display control unit that displays a composite image obtained by combining the first image and the second image in a ratio obtained using information regarding at least a portion of the region of the first image on a display unit.

[0011] In addition, a medical image processing method according to another embodiment of the present invention includes obtaining a first image, which is a medical image of a specific part of a subject, generating a second image from the first image, the second image having higher image quality than the first image, using an image quality improvement engine including a machine learning engine, and displaying a composite image obtained by combining the first image and the second image in a ratio obtained using information regarding at least a portion of the region of the first image on a display unit. Effect of the Invention

[0012] According to one aspect of the present invention, it is possible to generate an image that is more suitable for image diagnosis than in the past. [Brief description of the drawings]

[0013] [Figure 1] 1 shows an example of the configuration of a neural network related to image quality improvement processing. [Diagram 2] 1 shows an example of the configuration of a neural network related to a shooting location estimation process. [Diagram 3] 1 shows an example of the configuration of a neural network related to image authenticity evaluation processing. [Figure 4] 1 illustrates an example of a schematic configuration of an image processing device according to a first embodiment. [Diagram 5] FIG. 4 is a flowchart showing an example of the flow of image processing according to the first embodiment. [Figure 6] FIG. 11 is a flowchart showing another example of the flow of image processing according to the first embodiment. [Figure 7]FIG. 11 is a flowchart showing an example of the flow of image processing according to the second embodiment. [Figure 8] FIG. 13 is a diagram for explaining image processing according to a fourth embodiment. [Figure 9] FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to a fourth embodiment. [Figure 10] FIG. 13 is a diagram for explaining image processing according to the fifth embodiment. [Figure 11] FIG. 13 is a flowchart showing an example of the flow of image quality improvement processing according to the fifth embodiment. [Figure 12] FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. [Figure 13] FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the sixth embodiment. [Figure 14] FIG. 13 is a diagram for explaining image processing according to the sixth embodiment. [Figure 15] 13 shows an example of a schematic configuration of an image processing device according to a seventh embodiment. [Figure 16] FIG. 13 is a flowchart showing an example of the flow of image processing according to the seventh embodiment. [Figure 17] 13 shows an example of a user interface according to the seventh embodiment. [Figure 18] 13 shows an example of a schematic configuration of an image processing device according to a ninth embodiment. [Figure 19] FIG. 13 is a flowchart showing an example of the flow of image processing according to the ninth embodiment. [Figure 20] 23 shows an example of a schematic configuration of an image processing device according to a twelfth embodiment. [Figure 21A] FIG. 23 is a flowchart showing an example of the flow of image quality improvement processing according to the thirteenth embodiment. [Figure 21B] FIG. 23 is a flowchart showing another example of the flow of the image quality improvement processing according to the thirteenth embodiment. [Figure 22] 23 shows an example of a schematic configuration of an image processing device according to a seventeenth embodiment. [Diagram 23] FIG. 23 is a flowchart showing an example of the flow of image processing according to the seventeenth embodiment. [Figure 24] 1 shows an example of the configuration of a neural network related to image quality improvement processing. [Diagram 25] 23 shows an example of a schematic configuration of an image processing device according to a nineteenth embodiment. [Figure 26] FIG. 23 is a flowchart showing an example of the flow of image processing according to the nineteenth embodiment. [Figure 27] FIG. 21 is a flow diagram showing an example of the flow of image processing according to the twenty-first embodiment. [Figure 28] 1 shows an example of a teacher image related to image quality improvement processing. [Figure 29] 1 shows an example of an input image for image quality improvement processing. [Diagram 30] 23 shows an example of a schematic configuration of an image processing device according to a 22nd embodiment. [Diagram 31] FIG. 22 is a flow diagram showing an example of the flow of image processing according to the twenty-second embodiment. [Diagram 32] FIG. 22 is a diagram for explaining a wide-angle image according to the twenty-second embodiment. [Diagram 33] FIG. 23 is a diagram for explaining image quality improvement processing according to the twenty-third embodiment. [Diagram 34] 23 shows an example of a user interface according to the twenty-fourth embodiment. [Diagram 35] 25 illustrates an example of a schematic configuration of an image processing device according to a 25th embodiment. [Diagram 36] 13 shows an example of the configuration of a neural network used as a machine learning engine according to Modification 6. [Figure 37] 13 shows an example of the configuration of a neural network used as a machine learning engine according to Modification 6. [Figure 38] 23 shows an example of a user interface according to the twenty-fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Exemplary embodiments for carrying out the present invention will now be described in detail with reference to the accompanying drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed according to the configuration or various conditions of the device to which the present invention is applied. In addition, in the drawings, the same reference numerals are used between the drawings to indicate elements that are identical or functionally similar.

[0015] <Terminology explanation> First, the terms used in this specification will be explained.

[0016] In the network herein, each device may be connected by a wired or wireless line. Here, the lines connecting the devices in the network include, for example, dedicated lines, local area network (hereinafter referred to as LAN) lines, wireless LAN lines, Internet lines, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like.

[0017] The medical image processing device may be composed of two or more devices capable of communicating with each other, or may be composed of a single device. Each component of the medical image processing device may be composed of a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Each component may be composed of a circuit that performs a specific function such as an ASIC. Each component may be composed of a combination of any other hardware and any software.

[0018] Moreover, the medical images processed by the medical image processing apparatus or medical image processing method according to the following embodiments include images acquired using any modality (imaging apparatus, imaging method). The medical images to be processed may include medical images acquired by any imaging device or the like, or images created by a medical image processing device or medical image processing method according to the following embodiments.

[0019] Furthermore, the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. The medical image may also be an image that represents the structure (shape) of the predetermined part, or an image that represents its function. The image that represents the function includes, for example, an image that represents blood flow dynamics (blood flow rate, blood flow velocity, etc.), such as an OCTA image, a Doppler OCT image, an fMRI image, and an ultrasonic Doppler image. The predetermined part of the subject may be determined according to the subject to be photographed, and includes any part, such as the human eye (subject's eye), organs such as the brain, lungs, intestines, heart, pancreas, kidneys, and liver, the head, chest, legs, and arms.

[0020] The medical image may be a tomographic image of the subject or a front image. The front image includes, for example, a front image of the fundus, a front image of the anterior eye, a fundus image photographed by fluorescence, and an En-Face image generated by using data acquired by OCT (three-dimensional OCT data) of at least a part of the range in the depth direction of the subject. The En-Face image may be an OCTA En-Face image (motion contrast front image) generated by using data of at least a part of the range in the depth direction of the subject of three-dimensional OCTA data (three-dimensional motion contrast data). The three-dimensional OCT data and the three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0021] Moreover, the imaging device is a device for capturing an image used in diagnosis. The imaging device includes, for example, a device for obtaining an image of a predetermined part of a subject by irradiating the predetermined part with light, radiation such as X-rays, electromagnetic waves, or ultrasound, or a device for obtaining an image of the predetermined part by detecting radiation emitted from a subject. More specifically, the imaging device according to the following embodiments includes at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

[0022] The OCT device may include a time domain OCT (TD-OCT) device or a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device or a swept wavelength OCT (SS-OCT) device. The SLO device or OCT device may include a wavefront compensation SLO (AO-SLO) device or a wavefront compensation OCT (AO-OCT) device using a wavefront compensation optical system. The SLO device or OCT device may include a polarized SLO (PS-SLO) device or a polarized OCT (PS-OCT) device for visualizing information related to polarization phase difference and depolarization.

[0023] The image management system is a device and a system that receives and stores images captured by an imaging device and images that have been processed. The image management system can also transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS). In particular, the image management system according to the following embodiment includes a database that can store various information such as information on subjects associated with received images and imaging times. The image management system is also connected to a network, and can transmit and receive images, convert images, and transmit and receive various information associated with stored images in response to a request from other devices.

[0024] The imaging conditions are various information at the time of imaging of an image acquired by an imaging device. The imaging conditions include, for example, information on the imaging device, information on the facility where imaging was performed, information on the examination related to imaging, information on the photographer, and information on the subject. The imaging conditions also include, for example, imaging date and time, name of the imaging part, imaging area, imaging angle of view, imaging method, image resolution and gradation, image size, applied image filter, information on the data format of the image, and information on the radiation dose. The imaging area may include a peripheral area shifted from a specific imaging part, an area including multiple imaging parts, and the like.

[0025] The photographing conditions can be stored in a data structure constituting an image, or can be stored as photographing condition data separate from the image, or can be stored in a database or image management system related to the photographing device. Therefore, the photographing conditions can be acquired by a procedure corresponding to a storage means of the photographing conditions of the photographing device. Specifically, the photographing conditions can be acquired, for example, by analyzing the data structure of the image output by the photographing device, acquiring photographing condition data corresponding to the image, or accessing an interface for acquiring the photographing conditions from a database related to the photographing device.

[0026] Depending on the imaging device, there may be imaging conditions that cannot be acquired because they are not saved. For example, the imaging device may not have a function to acquire or save a specific imaging condition, or such a function may be disabled. In addition, for example, imaging conditions may not be saved because they are not related to the imaging device or imaging. Furthermore, for example, imaging conditions may be hidden, encrypted, or cannot be acquired without the right. However, even imaging conditions that are not saved may be acquired. For example, image analysis may be performed to identify the imaging part name and imaging area.

[0027] A machine learning model is a model that has been trained (learned) in advance using appropriate teacher data (learning data) for an arbitrary machine learning algorithm. The teacher data is composed of one or more pairs of input data and output data (correct answer data). The format and combination of the input data and output data of the pairs that make up the teacher data may be suitable for a desired configuration, such as one being an image and the other being a numerical value, one being composed of a group of multiple images and the other being a character string, or both being images.

[0028] Specifically, for example, there is training data (hereinafter, referred to as first training data) configured by a group of pairs of an image acquired by OCT and an imaging part label corresponding to the image. The imaging part label is a unique numerical value or character string that represents the part. Another example of training data is training data (hereinafter, referred to as second training data) configured by a group of pairs of a noisy low-quality image acquired by normal OCT imaging and a high-quality image that has been imaged multiple times by OCT and processed to improve image quality.

[0029] When input data is input to a machine learning model, output data according to the design of the machine learning model is output. The machine learning model outputs output data that is likely to correspond to the input data, for example, according to a trend trained using teacher data. The machine learning model can also output the possibility of each type of output data corresponding to the input data as a numerical value, for example, according to a trend trained using teacher data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with first teacher data, the machine learning model outputs the imaging site label of the imaging site captured in the image, or outputs the probability for each imaging site label. Also, for example, when a noisy low-quality image acquired by normal OCT imaging is input to a machine learning model trained with second teacher data, the machine learning model outputs a high-quality image equivalent to an image captured multiple times by OCT and processed for high image quality. Note that the machine learning model can be configured not to use the output data it outputs as teacher data from the viewpoint of maintaining quality.

[0030] The machine learning algorithm also includes a deep learning method such as a convolutional neural network (CNN). In a deep learning method, if the parameter settings for the layers and nodes constituting the neural network are different, the degree to which the tendency trained using the teacher data can be reproduced in the output data may differ. For example, in a deep learning machine learning model using the first teacher data, if more appropriate parameters are set, the probability of outputting a correct imaging region label may be higher. Also, for example, in a deep learning machine learning model using the second teacher data, if more appropriate parameters are set, an image with higher image quality may be output.

[0031] Specifically, the parameters in the CNN can include, for example, the filter kernel size, the number of filters, the stride value, and the dilation value set for the convolution layer, as well as the number of nodes output by the fully connected layer. The parameter group and the number of training epochs can be set to values ​​that are suitable for the usage form of the machine learning model based on the teacher data. For example, the parameter group and the number of epochs can be set based on the teacher data to output the correct imaging site label with a higher probability or output a higher quality image.

[0032] Here is an example of a method for determining such a parameter set and the number of epochs. First, 70% of the pair sets constituting the teacher data are set for training, and the remaining 30% are set for evaluation, randomly. Next, the machine learning model is trained using the pair sets for training, and at the end of each epoch of training, a training evaluation value is calculated using the pair sets for evaluation. The training evaluation value is, for example, the average value of a group of values ​​obtained by evaluating the output when the input data constituting each pair is input to the machine learning model being trained, and the output data corresponding to the input data, using a loss function. Finally, the parameter set and the number of epochs when the training evaluation value is the smallest are determined as the parameter set and the number of epochs of the machine learning model. In this way, by dividing the pair sets constituting the teacher data into those for training and those for evaluation and determining the number of epochs, it is possible to prevent the machine learning model from overlearning the pair sets for training.

[0033] The high-quality image engine (trained model for high-quality image improvement) is a module that improves the quality of an input low-quality image and outputs a high-quality image. In this specification, high-quality image improvement refers to converting an input image into an image with a quality more suitable for image diagnosis, and a high-quality image refers to an image converted into an image with a quality more suitable for image diagnosis. In addition, a low-quality image is, for example, a two-dimensional image or a three-dimensional image acquired by X-ray photography, CT, MRI, OCT, PET, or SPECT, or a three-dimensional moving image of a CT taken continuously without being set to have a particularly high quality. Specifically, low-quality images include, for example, images acquired by low-dose photography using an X-ray imaging device or CT, photography using an MRI without using a contrast agent, images acquired by short-time photography using OCT, and OCTA images acquired with a small number of photography sessions.

[0034] In addition, the content of image quality suitable for image diagnosis depends on what is to be diagnosed by various image diagnoses. Therefore, it is not possible to make a general statement, but for example, image quality suitable for image diagnosis includes image quality with little noise, high contrast, displaying the captured subject in colors and gradations that are easy to observe, large image size, and high resolution. It can also include image quality in which non-existent objects and gradations that are drawn in the image generation process are removed from the image.

[0035] In addition, when high-quality images with low noise and high contrast are used for image analysis such as blood vessel analysis processing of images such as OCTA, or area segmentation processing of images such as CT or OCT, analysis can often be performed more accurately than when low-quality images are used. Therefore, high-quality images output by the image quality improvement engine can be useful not only for image diagnosis but also for image analysis.

[0036] In the image processing method constituting the image quality improvement method in the following embodiment, processing using various machine learning algorithms such as deep learning is performed. Note that in addition to processing using the machine learning algorithm, the image processing method may perform any existing processing such as various image filter processing, matching processing using a database of high quality images corresponding to similar images, and knowledge-based image processing.

[0037] In particular, as an example of a CNN configuration for improving the image quality of a two-dimensional image, there is a configuration shown in FIG. 1. The CNN configuration includes a group of multiple convolution processing blocks 100. The convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a rectifier linear unit. The CNN configuration also includes a merger layer 104 and a final convolution layer 105. The merger layer 104 merges the output value group of the convolution processing block 100 with the pixel value group constituting the image by concatenating or adding them. The final convolution layer 105 outputs the pixel value group constituting the high-image-quality image Im120 merged in the merger layer 104. In this configuration, a group of pixel values ​​constituting an input image Im110 is output through the group of convolution processing blocks 100, and the group of pixel values ​​constituting the input image Im110 is synthesized in the synthesis layer 104. The synthesized group of pixel values ​​is then formed into a high-quality image Im120 in the final convolution layer 105.

[0038] For example, a certain degree of image quality improvement can be achieved by setting the number of convolution processing blocks 100 to 16, setting the filter kernel size to 3 pixels wide and 3 pixels high, and setting the number of filters to 64 as parameters of the convolution layer 101 group. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using training data according to the usage form of the machine learning model. When a three-dimensional image or a four-dimensional image needs to be processed, the filter kernel size may be expanded to three dimensions or four dimensions.

[0039] It should be noted that when using some image processing methods, such as image processing using CNN, attention should be paid to the image size. Specifically, it should be noted that in order to address the problem of the peripheral areas of a high-quality image not being of sufficient quality, different image sizes may be required for the input low-quality image and the output high-quality image.

[0040] For the sake of clarity, this will not be specified in the following embodiments, but when a high-image-quality engine is adopted that requires different image sizes for the image input to the high-image-quality engine and the image output, the image size is adjusted appropriately. Specifically, the image size is adjusted by padding an input image, such as an image used as teacher data for training a machine learning model or an image input to the high-image-quality engine, or by combining a shooting area around the input image. Note that the area to be padded is filled with a certain pixel value, filled with a neighboring pixel value, or mirror padded according to the characteristics of the high-image-quality technique so that the image quality can be effectively improved.

[0041] The image quality improvement method may be implemented using only one image processing method, or may be implemented using a combination of two or more image processing methods. A plurality of image quality improvement methods may be implemented in parallel to generate a plurality of high-quality image groups, and the highest quality image may be selected as the final high-quality image. The highest quality image may be selected automatically using an image quality assessment index, or may be selected according to an instruction from an examiner (user) by displaying a plurality of high-quality image groups on a user interface provided on an arbitrary display unit or the like.

[0042] In addition, since an input image that has not been enhanced in image quality may be more suitable for image diagnosis, the input image may be included in the final image selection. In addition, parameters may be input to the image enhancement engine together with the low image quality image. For example, parameters specifying the degree of image enhancement or parameters specifying the image filter size used in the image processing method may be input to the image enhancement engine together with the input image.

[0043] The imaging location estimation engine is a module that estimates the imaging site and imaging area of ​​an input image. The imaging location estimation engine can output the location of the imaging site and imaging area depicted in the input image, or the probability that the imaging site and imaging area are the imaging site and imaging area for each imaging site label and imaging area label of a required level of detail.

[0044] Depending on the imaging device, the imaging part and imaging area may not be saved as imaging conditions, or may not be acquired by the imaging device and saved. Even if the imaging part and imaging area are saved, the imaging part and imaging area may not be saved at the required level of detail. For example, the imaging part may only be saved as "posterior eye", and it is not clear whether it is "macular", "optic disc", "macular and optic disc", or "other". In another example, the imaging part may only be saved as "breast", and it is not clear whether it is "right breast", "left breast", or "both". Therefore, by using the imaging part estimation engine, it is possible to estimate the imaging part and imaging area of ​​the input image in these cases.

[0045] The image and data processing method constituting the estimation method of the imaging location estimation engine performs processing using various machine learning algorithms such as deep learning. Note that the image and data processing method may perform any existing estimation processing such as natural language processing, matching processing using a database of similar images and similar data, knowledge base processing, etc. in addition to or instead of the processing using the machine learning algorithm. Note that the teacher data for training the machine learning model constructed using the machine learning algorithm may be images labeled with the imaging site and imaging area. In this case, the image of the teacher data is used as input data, and the labels of the imaging site and imaging area are used as output data.

[0046] In particular, an example of a CNN configuration for estimating the shooting location of a two-dimensional image is the configuration shown in Fig. 2. The CNN configuration includes a group of multiple convolution processing blocks 200, each of which is made up of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a normalized linear function. The CNN configuration also includes a final convolutional layer 204, a fully connected layer 205, and an output layer 206. The fully connected layer 205 fully connects the output value group of the convolution processing block 200. The output layer 206 also uses a Softmax function to output the probability of each expected imaging region label for the input image Im210 as an estimation result (Result) 207. In this configuration, for example, if the input image Im210 is an image of the "macular region," the highest probability is output for the imaging region label corresponding to the "macular region."

[0047] For example, the imaging site can be estimated with a certain degree of accuracy by setting the number of convolution processing blocks 200 to 16, the parameters of the convolution layer 201 group to a filter kernel size of 3 pixels wide and 3 pixels high, and the number of filters to 64. However, in practice, as described in the above description of the machine learning model, a better parameter group can be set using teacher data according to the usage form of the machine learning model. When a three-dimensional image or a four-dimensional image needs to be processed, the filter kernel size may be expanded to three dimensions or four dimensions. The estimation method may be implemented using only one image and data processing method, or may be implemented using a combination of two or more images and data processing methods.

[0048] The image quality assessment engine is a module that outputs an image quality assessment index for an input image. The image quality assessment processing method for calculating the image quality assessment index performs processing using various machine learning algorithms such as deep learning. The image quality assessment processing method may perform any existing evaluation processing such as an image noise measurement algorithm and a matching process using a database of image quality assessment indices corresponding to similar images and base images. These evaluation processes may be performed in addition to or instead of processing using a machine learning algorithm.

[0049] For example, the image quality assessment index can be obtained from a machine learning model constructed using a machine learning algorithm. In this case, the pair of input data constituting the teacher data for training the machine learning model is an image group consisting of a low image quality image group and a high image quality image group captured in advance under various shooting conditions. Moreover, the pair of output data constituting the teacher data for training the machine learning model is, for example, a group of image quality assessment indices set for each of the image groups of the input data by an examiner performing image diagnosis.

[0050] The authenticity evaluation engine in the description of the present invention is a module that evaluates the drawing of an input image and evaluates with a certain degree of accuracy whether the image was captured and acquired by the target imaging device. In the authenticity evaluation processing method, processing using various machine learning algorithms such as deep learning is performed. Note that in the authenticity evaluation processing method, in addition to or instead of the processing using the machine learning algorithm, any existing evaluation processing such as knowledge-based processing may be performed.

[0051] For example, the authenticity evaluation process can be performed by a machine learning model constructed using a machine learning algorithm. First, the training data of the machine learning model will be described. The training data includes a pair group of high-quality images captured in advance under various shooting conditions and a label (hereinafter, genuine label) indicating that the images were captured and acquired by the target imaging device. The training data also includes a pair group of high-quality images generated by inputting low-quality images to an image quality improvement engine (first-level image quality improvement engine) and a label (hereinafter, counterfeit label) indicating that the images were not captured and acquired by the target imaging device. A machine learning model trained using such training data outputs a counterfeit label when a high-quality image generated by the first-level image quality improvement engine is input.

[0052] In particular, an example of the configuration of a CNN that performs authenticity evaluation processing of a two-dimensional image is shown in FIG. 3. The CNN configuration includes a group of multiple convolution processing blocks 300, each of which is composed of a convolution layer 301, a batch normalization layer 302, and an activation layer 303 using a normalized linear function. The CNN configuration also includes a final convolution layer 304, a fully connected layer 305, and an output layer 306. The fully connected layer 305 fully connects the output value groups of the convolution processing blocks 300. The output layer 306 also uses a Sigmoid function to output a value of 1 (true) representing an authentic label or a value of 0 (false) representing a counterfeit label as the result (Result) 307 of the authenticity evaluation processing for the input image Im 310.

[0053] In addition, by setting the number of convolution processing blocks 300 to 16, the parameters of the convolution layer 301 group, the filter kernel size to 3 pixels wide and 3 pixels high, and the number of filters to 64, a correct authenticity evaluation process result can be obtained with a certain degree of accuracy. However, as described in the above description of the machine learning model, in practice, a better parameter group can be set using training data according to the usage form of the machine learning model. In addition, when it is necessary to process three-dimensional or four-dimensional images, the filter kernel size may be expanded to three or four dimensions.

[0054] The authenticity evaluation engine may output an authenticity label when a high-quality image generated by a high-quality engine (second-level image quality engine) that improves image quality more than the first-level image quality engine is input. In other words, the authenticity evaluation engine cannot reliably evaluate whether an input image is an image that was photographed and acquired by a photographing device, but it can evaluate whether the image resembles an image that was photographed and acquired by a photographing device. By utilizing this characteristic, by inputting a high-quality image generated by a high-quality engine to the authenticity evaluation engine, it is possible to evaluate whether the high-quality image generated by the high-quality engine has been sufficiently improved in image quality.

[0055] The efficiency and accuracy of both engines may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine in cooperation with each other. In this case, first, the machine learning model of the image quality improvement engine is trained so that when the authenticity evaluation engine evaluates a high-quality image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image generated by the image quality improvement engine, an authenticity label is output. In parallel, the machine learning model of the authenticity evaluation engine is trained so that when the authenticity evaluation engine evaluates an image acquired by a photographing device, an authenticity label is output. This improves the efficiency and accuracy of the image quality improvement engine and the authenticity evaluation engine.

[0056] <First embodiment> The medical image processing apparatus according to the first embodiment will be described below with reference to Figures 4 and 5. Figure 4 shows an example of a schematic configuration of the image processing apparatus according to this embodiment.

[0057] The image processing device 400 is connected to the photographing device 10 and the display unit 20 via a circuit or a network. The photographing device 10 and the display unit 20 may also be directly connected. Although these devices are separate devices in this embodiment, some or all of these devices may be configured as an integrated device. These devices may also be connected to any other device via a circuit or a network, or may be configured as an integrated device with any other device.

[0058] The image processing device 400 includes an acquisition unit 401, a shooting condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). The image processing device 400 may be composed of a plurality of devices each including some of these components. The acquisition unit 401 can acquire various data and images from the imaging device 10 or other devices, and can acquire input from an examiner via an input device (not shown). The input device may be a mouse, a keyboard, a touch panel, or any other input device. The display unit 20 may be configured as a touch panel display.

[0059] The imaging condition acquisition unit 402 acquires imaging conditions of the medical image (input image) acquired by the acquisition unit 401. Specifically, the imaging condition group stored in the data structure constituting the medical image is acquired according to the data format of the medical image. If the imaging condition is not stored in the medical image, the imaging information group including the imaging condition group can be acquired via the acquisition unit 401 from the imaging device 10 or the image management system.

[0060] The image quality improvement possibility determination unit 403 determines whether or not the medical image can be handled by the image quality improvement unit 404 using the group of imaging conditions acquired by the imaging condition acquisition unit 402. The image quality improvement unit 404 improves the image quality of the medical image that can be handled, and generates a high-image quality image suitable for image diagnosis. The output unit 405 displays the high-image quality image generated by the image quality improvement unit 404, the input image, various information, etc. on the display unit 20. The output unit 405 may also store the generated high-image quality image, etc. in a storage device (storage unit) connected to the image processing device 400.

[0061] Next, a detailed description will be given of the image quality improvement unit 404. The image quality improvement unit 404 is provided with an image quality improvement engine. The image quality improvement method provided in the image quality improvement engine according to this embodiment performs processing using a machine learning algorithm.

[0062] In this embodiment, training of a machine learning model related to a machine learning algorithm uses teacher data consisting of a pair group of input data, which is a low-quality image having a specific imaging condition assumed to be a processing target, and output data, which is a high-quality image corresponding to the input data. Note that the specific imaging condition specifically includes a predetermined imaging part, imaging method, imaging angle of view, image size, etc.

[0063] In this embodiment, the input data of the teacher data is a low-quality image acquired by the same model and the same settings as the photographing device 10. The output data of the teacher data is a high-quality image acquired by the settings and image processing of the same model as the photographing device 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by performing superimposition processing such as averaging on a group of images (original images) acquired by photographing multiple times. Here, the high-quality image and the low-quality image will be described using the motion contrast data of OCTA as an example. Here, the motion contrast data is data used in OCTA and the like, which is obtained by repeatedly photographing the same location of the photographing target and detecting the temporal change of the photographing target between the photographing. At this time, the En-Face image (motion contrast front image) of OCTA can be generated by generating a front image using data of a desired range in the depth direction of the photographing target from the calculated motion contrast data (an example of three-dimensional medical image data). In the following, repeatedly photographing OCT data at the same location is called NOR (Number Of Repeat).

[0064] In this embodiment, two different methods for generating high-quality images and low-quality images by overlay processing will be described with reference to FIG.

[0065] The first method will be described with reference to FIG. 28(a) with respect to motion contrast data generated from OCT data obtained by repeatedly capturing the same location of a subject as an example of a high-quality image. In FIG. 28(a), Im2810 indicates three-dimensional motion contrast data, and Im2811 indicates two-dimensional motion contrast data constituting the three-dimensional motion contrast data. Im2811-1 to Im2811-3 indicate OCT tomographic images (B-scans) for generating Im2811. Here, NOR indicates the number of OCT tomographic images in Im2811-1 to Im2811-3 in FIG. 28(a), and NOR is 3 in the example shown in the figure. Im2811-1 to Im2811-3 are captured at a predetermined time interval (Δt). The same location indicates one line in the front direction (XY) of the subject's eye, and corresponds to the location of Im2811 in FIG. 28(a). The front direction is an example of a direction intersecting with the depth direction. Since the motion contrast data is data that detects a change over time, it is necessary to set NOR at least twice to generate this data. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, two data are generated when the motion contrast data is generated only by OCT of adjacent time intervals (first and second, second and third), respectively. When the motion contrast data is generated using OCT data of distant time intervals (first and third), a total of three data are generated. That is, when NOR is increased to three, four, etc., the number of motion contrast data at the same location also increases. High-quality motion contrast data can be generated by aligning multiple motion contrast data obtained by repeatedly photographing the same location and performing superposition processing such as averaging. Therefore, NOR should be at least three times or more, and preferably five times or more. On the other hand, an example of a low-quality image corresponding to this is the motion contrast data before superposition processing such as averaging. In this case, it is desirable to use the low-quality image as a reference image when performing superimposition processing such as averaging.When performing the overlay process, if the position and shape of the target image are deformed relative to the reference image to perform alignment, there is almost no spatial misalignment between the reference image and the image after the overlay process. Therefore, it is easy to pair low-quality images with high-quality images. Note that the target image that has been subjected to the image deformation process for alignment, rather than the reference image, may be used as the low-quality image. By using each of the original image groups (reference image and target image) as input data and the corresponding overlaid images as output data, it is possible to generate multiple pair groups. For example, when obtaining one overlaid image from a group of 15 original images, it is possible to generate a pair of the first original image and the overlaid image from the original image group, and a pair of the second original image and the overlaid image from the original image group. In this way, when obtaining one overlaid image from a group of 15 original images, it is possible to generate 15 pair groups of one image from the original image group and the overlaid image. Note that three-dimensional high-quality data can be generated by repeatedly photographing the same location in the main scanning (X) direction and scanning while shifting it in the sub-scanning (Y) direction.

[0066] The second method is a process of generating a high-quality image by superimposing motion contrast data obtained by photographing the same region of the subject multiple times, and is described with reference to FIG. 28(b). The same region refers to an area of ​​3×3 mm or 10×10 mm in the front direction (XY) of the subject's eye, and means obtaining three-dimensional motion contrast data including the depth direction of the tomographic image. When photographing the same region multiple times and performing superimposing, it is desirable to perform NOR two or three times in order to shorten each photographing. In addition, in order to generate high-quality three-dimensional motion contrast data, at least two or more three-dimensional data of the same region are obtained. FIG. 28(b) shows an example of multiple three-dimensional motion contrast data. Im2820 to Im2840 are three-dimensional motion contrast data as described in FIG. 28(a). Using these two or more three-dimensional motion contrast data, alignment processing is performed in the front direction (XY) and the depth direction (Z), and after removing data that becomes artifacts in each data, averaging processing is performed. This makes it possible to generate one piece of high-quality three-dimensional motion contrast data from which artifacts have been removed. A high-quality image is obtained by generating an arbitrary plane from the three-dimensional motion contrast data. On the other hand, it is preferable that the corresponding low-quality image is an arbitrary plane generated from reference data when performing superposition processing such as averaging. As explained in the first method, since there is almost no spatial positional deviation between the reference image and the image after averaging, a low-quality image and a high-quality image can be easily paired. Note that the low-quality image may be an arbitrary plane generated from target data that has been subjected to image deformation processing for alignment, rather than the reference data.

[0067] The first method places less of a burden on the subject because the image is taken in one go. However, the more NORs there are, the longer the image acquisition time becomes. Also, if artifacts such as eye clouding or eyelashes appear during the image acquisition, it is not always possible to obtain a good image. The second method places a little more strain on the subject because multiple shots are taken. However, each shot takes less time, and even if an artifact appears in one shot, if the artifact is not visible in another shot, a clean image with fewer artifacts can ultimately be obtained. Taking these features into account, a method can be selected according to the subject's situation when collecting data.

[0068] In this embodiment, the motion contrast data is used as an example, but the present invention is not limited thereto. Since the OCT data is captured to generate the motion contrast data, the above method can be used for the OCT data as well. Furthermore, although the tracking process is not described in this embodiment, it is desirable to capture the image while tracking the subject's eye in order to capture the same location or region of the subject's eye.

[0069] In this embodiment, since a pair of three-dimensional high-quality data and low-quality data is created, an arbitrary pair of two-dimensional images can be generated from the pair. This will be described with reference to FIG. 29. For example, when the target image is an OCTA En-Face image, the OCTA En-Face image is generated in a desired depth range from the three-dimensional data. The desired depth range refers to the range in the Z direction in FIG. 28. An example of the OCTA En-Face image generated here is shown in FIG. 29(a). As the OCTA En-Face image, learning is performed using OCTA En-Face images generated in different depth ranges such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940). Note that the types of OCTA En-Face images are not limited to these, and the types may be increased by generating OCTA En-Face images in which different depth ranges are set by changing the reference layer and offset value. When learning, OCTA En-Face images of different depths may be learned separately, or images of different depth ranges may be combined and learned (for example, divided into superficial and deep layers), or OCTA En-Face images of all depth ranges may be learned together. In the case of En-Face images of luminance generated from OCT data, learning is performed using multiple En-Face images generated from an arbitrary depth range, as with OCTA En-Face. For example, consider a case where the image quality improvement engine includes a machine learning engine obtained using learning data including multiple motion contrast front images corresponding to different depth ranges of the subject's eye. In this case, the acquisition unit can acquire a motion contrast front image corresponding to a part of a long depth range including different depth ranges as the first image. That is, a motion contrast front image corresponding to a depth range different from the multiple depth ranges corresponding to the multiple motion contrast front images included in the learning data can be used as an input image at the time of image quality improvement. Of course, a motion contrast front image of the same depth range as that at the time of learning may be used as an input image at the time of image quality improvement. Furthermore, some of the depth ranges may be set in response to the examiner pressing an arbitrary button on the user interface, or may be set automatically.It should be noted that the above is not limited to a motion contrast front image, but can also be applied to, for example, a luminance En-Face image.

[0070] When the image to be processed is a tomographic image, learning is performed using an OCT tomographic image, which is a B-scan, or a tomographic image of motion contrast data. This will be described with reference to FIG. 29(b). In FIG. 29(b), Im2951 to Im2953 are OCT tomographic images. The images are different in FIG. 29(b) because they show tomographic images at different positions in the sub-scanning (Y) direction. For tomographic images, learning may be performed together without considering the difference in position in the sub-scanning direction. However, in the case of images of different photographed parts (for example, the center of the macula, the center of the optic disc), learning may be performed separately for each part, or learning may be performed together without considering the photographed part. Note that image features are significantly different between OCT tomographic images and tomographic images of motion contrast data, so it is better to perform learning separately.

[0071] The superimposed image obtained by the superimposition process has pixels that are commonly depicted in the original images emphasized, resulting in a high-quality image suitable for image diagnosis. In this case, the generated high-quality image has high contrast with clear differences between low-luminance and high-luminance areas as a result of the emphasis on the commonly depicted pixels. In addition, for example, in the superimposed image, random noise that occurs with each shot can be reduced, and areas that were not well depicted in the original images at a certain point in time can be interpolated by other original images.

[0072] In addition, when the input data for the machine learning model needs to be composed of multiple images, the necessary number of original image groups can be selected from the original image group and used as input data. For example, when obtaining one overlaid image from a group of 15 original images, if two images are required as input data for the machine learning model, 105 (15C2=105) pair groups can be generated.

[0073] In addition, among the pairs constituting the teacher data, pairs that do not contribute to high image quality can be removed from the teacher data. For example, if the high-image-quality image that is the output data constituting the pair of the teacher data has an image quality that is not suitable for image diagnosis, the image output by the image quality improvement engine that has learned using the teacher data may also have an image quality that is not suitable for image diagnosis. Therefore, by removing pairs whose output data has an image quality that is not suitable for image diagnosis from the teacher data, the possibility that the image quality improvement engine will generate an image with an image quality that is not suitable for image diagnosis can be reduced.

[0074] In addition, if the average luminance or luminance distribution of a pair of images is significantly different, the image quality improvement engine that has learned using the training data may output an image that is not suitable for image diagnosis and has a luminance distribution that is significantly different from that of the low-quality image. For this reason, it is possible to remove pairs of input data and output data that have significantly different average luminance or luminance distribution from the training data.

[0075] Furthermore, if the structure or position of the object depicted in the paired image group is significantly different, the image quality improvement engine that has learned using the training data may output an image that is not suitable for image diagnosis, in which the object is depicted in a structure or position significantly different from that of the low-quality image. For this reason, it is also possible to remove from the training data pairs of input data and output data in which the structure or position of the object depicted is significantly different. In addition, from the viewpoint of maintaining quality, the image quality improvement engine can be configured not to use the high-quality image that it outputs as training data.

[0076] By using the image quality improvement engine that has performed machine learning in this way, when a medical image acquired in one shot is input, the image quality improvement unit 404 can output a high-image quality image in which contrast has been increased, noise has been reduced, etc., by overlay processing. Therefore, the image quality improvement unit 404 can generate a high-image quality image suitable for image diagnosis based on a low-image quality image that is an input image.

[0077] Next, a series of image processing according to this embodiment will be described with reference to the flow diagram of Fig. 5. Fig. 5 is a flow diagram of a series of image processing according to this embodiment. First, when a series of image processing according to this embodiment is started, the process proceeds to step S510.

[0078] In step S510, the acquisition unit 401 acquires an image captured by the photographing device 10 as an input image from the photographing device 10 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the photographing device 10. Such a request may be issued, for example, when the photographing device 10 generates an image, before or after the image generated by the photographing device 10 is stored in a storage device provided in the photographing device 10, when the stored image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0079] The acquisition unit 401 may acquire data for generating an image from the image capturing device 10, and acquire an image generated by the image processing device 400 based on the data as an input image. In this case, any existing image generation method may be adopted as the image generation method for the image processing device 400 to generate various images.

[0080] In step S520, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image. Specifically, the photographing condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. As described above, when the photographing conditions are not stored in the input image, the photographing condition acquisition unit 402 can acquire a photographing information group including the photographing condition group from the photographing device 10 or an image management system (not shown).

[0081] In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not the image quality of the input image can be improved by the image quality improvement engine provided in the image quality improvement unit 404. Specifically, the image quality improvement possibility determination unit 403 determines whether or not the shooting part, shooting method, shooting angle of view, and image size of the input image match the conditions that can be handled by the image quality improvement engine.

[0082] When the image quality improvement possibility determination unit 403 determines all the shooting conditions and determines that they are manageable, the process proceeds to step S540. On the other hand, when the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these shooting conditions, the process proceeds to step S550.

[0083] Depending on the setting or implementation of the image processing device 400, even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, imaging angle of view, and image size, the image quality improvement process in step S540 may be performed. For example, such processing may be performed when the image quality improvement engine is assumed to be comprehensively compatible with any imaging part of the subject and is implemented to be able to handle even if the input data includes an unknown imaging part. In addition, the image quality improvement possibility determination unit 403 may determine whether or not at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image matches a condition that can be handled by the image quality improvement engine according to a desired configuration.

[0084] In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image and generate a high-image quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, which generates a high-image quality image with improved image quality. The image quality improvement engine generates a high-image quality image as if it had been subjected to overlay processing using the input image, based on a machine learning model that has been subjected to machine learning using teacher data. Therefore, the image quality improvement engine can generate a high-image quality image with reduced noise and enhanced contrast compared to the input image.

[0085] Depending on the settings and implementation form of the image processing device 400, the image quality improvement unit 404 may input parameters to the image quality improvement engine together with the input image according to the group of shooting conditions to adjust the degree of image quality improvement, etc. Also, the image quality improvement unit 404 may input parameters according to the examiner's input to the image quality improvement engine together with the input image to adjust the degree of image quality improvement, etc.

[0086] In step S550, if a high-quality image has been generated in step S540, the output unit 405 outputs the high-quality image and displays it on the display unit 20. On the other hand, if it is determined in step S530 that high-quality image processing is not possible, the input image is output and displayed on the display unit 20. Note that instead of displaying the output image on the display unit 20, the output unit 405 may cause the image capture device 10 or another device to display or store the output image. Depending on the settings and implementation form of the image processing device 400, the output unit 405 may process the output image so that it can be used by the image capture device 10 or another device, or convert the data format so that it can be transmitted to an image management system or the like.

[0087] As described above, the image processing device 400 according to this embodiment includes an acquisition unit 401 and an image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) which is an image of a predetermined part of a subject. The image quality improvement unit 404 uses an image quality improvement engine including a machine learning engine to generate a high image quality image (second image) from the input image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image. The image quality improvement engine includes a machine learning engine that uses the image obtained by the overlay process as learning data.

[0088] With this configuration, the image processing device 400 according to this embodiment can output a high-quality image in which noise is reduced and contrast is enhanced from an input image. Therefore, the image processing device 400 can obtain images suitable for image diagnosis, such as clearer images and images in which the part or lesion to be observed is emphasized, without increasing the invasiveness or labor of the photographer or subject, and at a lower cost than in the past.

[0089] The image processing device 400 further includes an image quality improvement possibility determination unit 403 that determines whether a high image quality image can be generated from an input image using an image quality improvement engine. The image quality improvement possibility determination unit 403 performs this determination based on at least one of the imaging part, imaging method, imaging angle of view, and image size of the input image.

[0090] With this configuration, the image processing device 400 according to this embodiment can exclude input images that cannot be processed by the image quality improvement unit 404 from the image quality improvement process, thereby reducing the processing load of the image processing device 400 and the occurrence of errors.

[0091] In this embodiment, the output unit 405 (display control unit) is configured to display the generated high-quality image on the display unit 20, but the operation of the output unit 405 is not limited to this. For example, the output unit 405 can also output the high-quality image to the photographing device 10 or other devices connected to the image processing device 400. Therefore, the high-quality image can be displayed on the user interface of these devices, saved in any storage device, used for any image analysis, or transmitted to an image management system.

[0092] In this embodiment, the image quality improvement possibility determination unit 403 determines whether or not an input image can be improved in image quality by the image quality improvement engine, and if the input image can be improved in image quality, the image quality improvement unit 404 improves the image quality. On the other hand, in cases where the image capturing device 10 captures images only under shooting conditions that allow for high image quality improvement, the image quality of the images acquired from the image capturing device 10 may be improved unconditionally. In this case, as shown in Fig. 6, the processes of steps S520 and S530 can be omitted, and step S540 can be performed following step S510.

[0093] In this embodiment, the output unit 405 is configured to display a high-quality image on the display unit 20. However, the output unit 405 may display a high-quality image on the display unit 20 in response to an instruction from the examiner. For example, the output unit 405 may display a high-quality image on the display unit 20 in response to the examiner pressing an arbitrary button on the user interface of the display unit 20. In this case, the output unit 405 may display the high-quality image in place of the input image, or may display the high-quality image alongside the input image.

[0094] Furthermore, when the output unit 405 displays a high-quality image on the display unit 20, the output unit 405 may display, together with the high-quality image, a display indicating that the displayed image is a high-quality image generated by processing using a machine learning algorithm. In this case, the user can easily identify from the display that the displayed high-quality image is not the image itself obtained by shooting, thereby reducing misdiagnosis and improving diagnostic efficiency. Note that the display indicating that the high-quality image is generated by processing using a machine learning algorithm may be in any form as long as it is a display that allows the input image and the high-quality image generated by the processing to be distinguished from each other.

[0095] Furthermore, the output unit 405 may display, on the display unit 20, a display indicating what kind of teacher data the machine learning algorithm has learned from, in relation to a display indicating that the image is a high-quality image generated by processing using a machine learning algorithm. Such a display may include an explanation of the types of input data and output data of the teacher data, or any display regarding the teacher data such as the imaging site included in the input data and output data.

[0096] In the image quality improvement engine according to the present embodiment, a superimposed image is used as output data of the teacher data, but the teacher data is not limited to this. As output data of the teacher data, a high-quality image obtained by performing at least one of a superimposition process, a group of processes described below, and an imaging method described below, which are means for obtaining a high-quality image, may be used.

[0097] For example, a high-quality image obtained by performing maximum a posteriori probability estimation processing (MAP estimation processing) on ​​a group of original images may be used as the output data of the training data. In the MAP estimation processing, a likelihood function is calculated from the probability density of each pixel value in multiple low-quality images, and the true signal value (pixel value) is estimated using the calculated likelihood function.

[0098] The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values ​​close to the true signal value. In addition, since the estimated signal value is calculated based on the probability density, randomly generated noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the image quality improvement engine can generate a high-quality image suitable for image diagnosis, in which noise is reduced and contrast is high, from the input image. Note that the method of generating pairs of input data and output data of the training data may be performed in the same manner as when the superimposed image is used as the training data.

[0099] Moreover, a high-quality image obtained by applying a smoothing filter process to an original image may be used as output data of the teacher data. In this case, the image quality improvement engine can generate a high-quality image in which random noise is reduced from the input image. Furthermore, an image obtained by applying a tone conversion process to an original image may be used as output data of the teacher data. In this case, the image quality improvement engine can generate a high-quality image in which contrast is enhanced from the input image. The method for generating pairs of input data and output data for the training data may be the same as that for the case where the superimposed image is used as the training data.

[0100] The input data of the teacher data may be an image acquired from an imaging device having the same image quality tendency as the imaging device 10. The output data of the teacher data may be a high-quality image acquired by a high-cost process such as an iterative approximation method, or a high-quality image acquired by imaging a subject corresponding to the input data with an imaging device having higher performance than the imaging device 10. Furthermore, the output data may be a high-quality image obtained by performing a rule-based noise reduction process. Here, the noise reduction process may include, for example, a process of replacing only one high-luminance pixel that appears in a low-luminance area and is clearly noise with an average value of nearby low-luminance pixel values. For this reason, the image quality improvement engine may use as learning data an image captured by a photographing device with higher performance than the photographing device used to capture the input image, or an image acquired in a photographing process with more steps than the photographing process of the input image. For example, when a motion contrast front image is used as an input image, the image quality improvement engine may use as learning data an image obtained by OCTA photography by an OCT photographing device with higher performance than the OCT photographing device used to capture the input image, or an image obtained in an OCTA photographing process with more steps than the OCTA photographing process of the input image.

[0101] Although not described in the present embodiment, a high-quality image generated from a plurality of images and used as output data for teacher data can be generated from a plurality of images that have already been aligned. The alignment process may involve, for example, selecting one of the plurality of images as a template, determining the similarity with other images while changing the position and angle of the template, determining the amount of misalignment with the template, and correcting each image based on the amount of misalignment. Any other existing alignment process may also be performed.

[0102] In addition, when aligning a three-dimensional image, the three-dimensional image may be decomposed into a plurality of two-dimensional images, and the two-dimensional images aligned for each may be integrated to align the three-dimensional image. Alternatively, the two-dimensional image may be decomposed into one-dimensional images, and the one-dimensional images aligned for each may be integrated to align the two-dimensional image. In addition, these alignments may be performed not on the images, but on data for generating the images.

[0103] In addition, in this embodiment, when the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement unit 404, the process proceeds to step S540, and the image quality improvement process by the image quality improvement unit 404 is started. In response to this, the output unit 405 may display the determination result by the image quality improvement possibility determination unit 403 on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner. At this time, the output unit 405 may display the input image and the imaging conditions such as the imaging part acquired for the input image, together with the determination result, on the display unit 20. In this case, the image quality improvement process is performed after the examiner determines whether the determination result is correct or not, so that the image quality improvement process based on an erroneous determination can be reduced.

[0104] In addition, instead of making a judgment by the image quality improvement feasibility judgment unit 403, the output unit 405 may display the input image and the shooting conditions such as the shooting part acquired for the input image on the display unit 20, and the image quality improvement unit 404 may start the image quality improvement process in response to an instruction from the examiner.

[0105] <Second embodiment> Next, an image processing apparatus according to a second embodiment will be described with reference to FIGS. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in the present embodiment, the image quality improvement unit includes multiple image quality improvement engines that perform machine learning using different teacher data, and generates multiple high-image-quality images for an input image.

[0106] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0107] The image quality improving unit 404 according to this embodiment is provided with two or more image quality improving engines that have been subjected to machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. Specifically, first, a group of pairs of an original image as input data and a superimposed image as output data, in which various imaging sites are imaged, is prepared. Next, the pair groups are grouped for each imaging site to create a teacher data group. For example, a teacher data group is created such that a first teacher data group is composed of a pair group acquired by imaging a first imaging site, and a second teacher data group is composed of a pair group acquired by imaging a second imaging site.

[0108] Then, machine learning is performed in separate image quality improvement engines using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, a second image quality improvement engine corresponding to a machine learning model trained with the second training data, and so on.

[0109] Since each of these image quality improvement engines uses different teacher data for training the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the imaging conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for the input image acquired by imaging the first imaging region, and a low degree of image quality improvement for the image acquired by imaging the second imaging region. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for the input image acquired by imaging the second imaging region, and a low degree of image quality improvement for the image acquired by imaging the first imaging region.

[0110] Since each of the teacher data is composed of a pair group grouped by the imaging site, the image quality tendencies of the image groups constituting the pair group are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image improvement engine according to the first embodiment for the corresponding imaging site. Note that the imaging conditions for grouping pairs of teacher data are not limited to the imaging site, and may be the imaging angle of view, the image resolution, or a combination of two or more of these.

[0111] A series of image processing according to this embodiment will be described below with reference to Fig. 7. Fig. 7 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S710 and S720 is similar to steps S510 and S520 according to the first embodiment, and therefore description thereof will be omitted. Note that, when unconditionally improving the image quality of an input image, after the processing of step S720, the processing of step S730 may be omitted and the processing may proceed to step S740.

[0112] When the shooting conditions of the input image are acquired in step S720, the process proceeds to step S730. In step S730, the image quality improvement possibility determination unit 403 uses the shooting conditions acquired in step S720 to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

[0113] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S760. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S740. Note that, depending on the settings and implementation form of the image processing device 400, step S740 may be performed, as in the first embodiment, even if the image quality improvement engines have determined that some shooting conditions cannot be handled.

[0114] In step S740, the image quality improving unit 404 selects an image quality improving engine that performs image quality improvement processing from the image quality improving engine group based on the imaging conditions of the input image acquired in step S720 and the teacher data information of the image quality improving engine group. Specifically, for example, for an imaging site in the imaging condition group acquired in step S720, an image quality improving engine that has teacher data information on the imaging site or surrounding imaging sites and provides a high level of image quality improvement is selected. In the above example, when the imaging site is the first imaging site, the image quality improving unit 404 selects the first image quality improving engine.

[0115] In step S750, image quality improvement unit 404 generates a high-image-quality image by improving the image quality of the input image using the image quality improvement engine selected in step S740. Thereafter, in step S760, if a high-image-quality image has been generated in step S750, output unit 405 outputs the high-image-quality image and displays it on display unit 20. On the other hand, if image quality improvement processing is determined to be impossible in step S730, output unit 405 outputs the input image and displays it on display unit 20. Note that when displaying the high-image-quality image on display unit 20, output unit 405 may display that the high-image-quality image was generated using the image quality improvement engine selected by image quality improvement unit 404.

[0116] As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine that corresponds to at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution of the input image.

[0117] With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

[0118] In this embodiment, the image quality improvement unit 404 selects an image quality improvement engine to be used for image quality improvement processing based on the photographing conditions of the input image, but the selection process of the image quality improvement engine is not limited to this. For example, the output unit 405 may display the photographing conditions of the acquired input image and the image quality improvement engine group on the user interface of the display unit 20, and the image quality improvement unit 404 may select an image quality improvement engine to be used for image quality improvement processing in response to an instruction from the examiner. The output unit 405 may display information on the teacher data used for learning each image quality improvement engine together with the image quality improvement engine group on the display unit 20. The display mode of the information on the teacher data used for learning the image quality improvement engine may be arbitrary, and for example, the image quality improvement engine group may be displayed using a name related to the teacher data used for learning.

[0119] Furthermore, the output unit 405 may display the image quality improvement engine selected by the image quality improvement unit 404 on the user interface of the display unit 20 and receive an instruction from the examiner. In this case, the image quality improvement unit 404 may determine, in response to the instruction from the examiner, whether or not to finally select the image quality improvement engine as the image quality improvement engine to be used for image quality improvement processing.

[0120] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0121] <Third embodiment> Next, an image processing apparatus according to a third embodiment will be described with reference to FIGS. In the first and second embodiments, the imaging condition acquisition unit 402 acquires an imaging condition group from the data structure of the input image, etc. In contrast, in the present embodiment, the imaging condition acquisition unit estimates the imaging part or imaging area of ​​the input image based on the input image using an imaging location estimation engine.

[0122] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the second embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the second embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configurations of the image processing devices according to the first and second embodiments, the configuration shown in FIG. 4 is indicated by the same reference numerals and description thereof will be omitted.

[0123] The imaging condition acquisition unit 402 according to this embodiment is provided with an imaging location estimation engine that estimates an imaging part or imaging area depicted in an input image acquired by the acquisition unit 401. In the imaging location estimation method provided in the imaging location estimation engine according to this embodiment, estimation processing is performed using a machine learning algorithm.

[0124] In this embodiment, training of a machine learning model related to a photographing location estimation method using a machine learning algorithm uses teacher data consisting of a pair group of input data, which is an image, and output data, which is a photographing part label or a photographing area label corresponding to the input data. Here, the input data refers to an image having a specific photographing condition assumed as a processing target (input image). The input data is preferably an image acquired from a photographing device having the same image quality tendency as the photographing device 10, and more preferably the same model with the same settings as the photographing device 10. The types of the photographing part label and the photographing area label, which are output data, may be the photographing part or the photographing area at least partially included in the input data. For example, in the case of OCT, the types of the photographing part label, which is output data, may be "macular area", "optic disc", "macular area and optic disc", "others", etc.

[0125] The imaging location estimation engine according to the present embodiment can output the location of the imaging part or imaging area depicted in the input image by performing learning using such teacher data. The imaging location estimation engine can also output the probability of the imaging part or imaging area for each imaging part label or imaging area label of a required level of detail. By using the imaging location estimation engine, the imaging condition acquisition unit 402 can estimate the imaging part or imaging area of ​​the input image based on the input image and acquire it as the imaging condition for the input image. Note that when the imaging location estimation engine outputs the probability of the imaging part or imaging area for each imaging part label or imaging area label, the imaging condition acquisition unit 402 acquires the imaging part or imaging area with the highest probability as the imaging condition of the input image.

[0126] Next, as in the second embodiment, a series of image processing steps according to this embodiment will be described with reference to the flow chart of Fig. 7. Note that the processing steps S710 and steps S730 to S760 according to this embodiment are similar to those in the second embodiment, and therefore will not be described. Note that, when unconditionally improving the image quality of an input image, after the processing step S720, the processing step S730 may be omitted and the processing may proceed to step S740.

[0127] When the input image is acquired in step S710, the process proceeds to step S720. In step S720, the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image acquired in step S710.

[0128] Specifically, the imaging condition group stored in the data structure constituting the input image is acquired according to the data format of the input image. Also, if the imaging condition group does not include information on the imaging part or imaging area, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine and estimates which imaging part the input image was acquired by imaging. Specifically, the imaging condition acquisition unit 402 inputs the input image to an imaging location estimation engine, evaluates the output probability for each imaging part label group, and sets and acquires the imaging part with the highest probability as the imaging condition of the input image.

[0129] In addition, when the input image does not store any shooting conditions other than the shooting part and shooting area, the shooting condition acquisition unit 402 can acquire a group of shooting information including a group of shooting conditions from the shooting device 10 or an image management system (not shown).

[0130] The subsequent processing is similar to the series of image processing according to the second embodiment, and therefore a description thereof will be omitted.

[0131] As described above, the photographing condition acquisition unit 402 according to this embodiment functions as an estimation unit that estimates at least one of the photographing part and the photographing area of ​​the input image. The photographing condition acquisition unit 402 includes a photographing location estimation engine that uses images labeled with the photographing part and the photographing area as learning data, and estimates the photographing part and the photographing area of ​​the input image by inputting the input image to the photographing location estimation engine.

[0132] In this way, the image processing device 400 according to this embodiment can acquire the imaging conditions for the imaging part or imaging area of ​​the input image based on the input image.

[0133] In this embodiment, the photographing condition acquisition unit 402 estimates the photographing part and the photographing area of ​​the input image using the photographing location estimation engine when the photographing condition group does not include information on the photographing part and the photographing area. However, the situation in which the photographing part and the photographing area are estimated using the photographing location estimation engine is not limited to this. The photographing condition acquisition unit 402 may estimate the photographing part and the photographing area using the photographing location estimation engine even when the information on the photographing part and the photographing area included in the data structure of the input image is insufficient as information at a required level of detail.

[0134] Furthermore, regardless of whether the data structure of the input image includes information on the photographed part and the photographed area, the photographing condition acquisition unit 402 may estimate the photographed part and the photographed area of ​​the input image using the photographing location estimation engine. In this case, the output unit 405 may display the estimation result output from the photographing location estimation engine and the information on the photographed part and the photographed area included in the data structure of the input image on the display unit 20, and the photographing condition acquisition unit 402 may determine these photographing conditions according to the examiner's instructions.

[0135] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0136] <Fourth embodiment> Next, an image processing device according to a fourth embodiment will be described with reference to Figures 4, 5, 8 and 9. In this embodiment, an image quality improvement unit enlarges or reduces an input image so that the input image has an image size that the image quality improvement engine can handle. The image quality improvement unit also reduces or enlarges an output image from the image quality improvement engine so that the image size of the output image becomes the image size of the input image, thereby generating a high-image quality image.

[0137] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0138] The image quality improvement unit 404 according to this embodiment is provided with an image quality improvement engine similar to the image quality improvement engine according to the first embodiment. However, in this embodiment, as teacher data used for learning the image quality improvement engine, a group of pairs of input data and output data, which are constituted by a group of images obtained by enlarging or reducing an image of the input data and an image of the output data to a certain image size, is used.

[0139] Now, referring to Fig. 8, the teacher data of the image quality improvement engine according to this embodiment will be described. As shown in Fig. 8, for example, consider the case where there is a low image quality image Im810 and a high image quality image Im820 that are smaller than a certain image size set for the teacher data. In this case, the low image quality image Im810 and the high image quality image Im820 are each enlarged so that they have the certain image size set for the teacher data. Then, the enlarged low image quality image Im811 and the enlarged high image quality image Im821 are paired, and the pair is used as one piece of teacher data.

[0140] As in the first embodiment, an image having a specific imaging condition assumed as a processing target (input image) is used as input data for the teacher data, and the specific imaging condition is a predetermined imaging part, imaging method, and imaging angle of view. In other words, unlike the first embodiment, the specific imaging condition according to this embodiment does not include an image size.

[0141] The image quality improvement unit 404 according to this embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-image quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image to a certain image size set for the teacher data, and inputs the deformed image to the image quality improvement engine. The image quality improvement unit 404 also reduces or enlarges an output image from the image quality improvement engine to the image size of the input image, generating a high-image quality image. Therefore, the image quality improvement unit 404 according to this embodiment can improve the image quality of an input image of an image size that could not be handled in the first embodiment by using the image quality improvement engine to generate a high-image quality image.

[0142] Next, a series of image processing according to this embodiment will be described with reference to Figs. 5 and 9. Fig. 9 is a flow diagram of image quality improvement processing according to this embodiment. Note that the processing of steps S510, S520, and S550 according to this embodiment is similar to these processing in the first embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than image size, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0143] In step S520, similarly to the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of imaging conditions to determine whether or not the high image quality engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement possibility determination unit 403 determines whether or not the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the high image quality engine can handle. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not determine the image size.

[0144] If the image quality improvement possibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

[0145] When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in Fig. 9 starts. In the image quality improvement process according to this embodiment, first, in step S910, image quality improvement unit 404 enlarges or reduces the input image to a certain image size set for the teacher data, and generates a deformed image.

[0146] Next, in step S920, the image quality improving unit 404 inputs the generated deformed image to an image quality improving engine to obtain a high-image-quality deformed image.

[0147] Thereafter, in step S930, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. When the image quality improvement unit 404 generates a high-quality image in step S930, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

[0148] As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of an input image to an image size that the image quality improvement engine can handle, and inputs the image to the image quality improvement engine. The image quality improvement unit 404 also generates a high-quality image by adjusting the output image from the image quality improvement engine to the original image size of the input image. As a result, the image processing device 400 according to this embodiment can use the image quality improvement engine to improve the image quality of input images of image sizes that could not be handled in the first embodiment, and generate high-quality images suitable for image diagnosis.

[0149] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0150] <Fifth embodiment> Next, an image processing device according to a fifth embodiment will be described with reference to Figures 4, 5, 10, and 11. In this embodiment, an image quality improvement unit generates a high-quality image through image quality improvement processing by an image quality improvement engine based on a certain resolution.

[0151] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0152] The image quality improving unit 404 according to this embodiment is equipped with an image quality improving engine similar to that of the first embodiment. However, in this embodiment, the teacher data used for learning the image quality improving engine is different from the teacher data in the first embodiment. Specifically, the image group constituting the pair group of input data and output data of the teacher data is enlarged or reduced to an image size such that the resolution of the image group becomes a certain resolution, and then padding is performed to obtain a sufficiently large certain image size. Here, the resolution of the image group refers to, for example, the spatial resolution of the imaging device or the resolution for the imaging area.

[0153] Here, the teacher data of the image quality improvement engine according to the present embodiment will be described with reference to Fig. 10. As shown in Fig. 10, for example, consider the case where there is a low-quality image Im1010 and a high-quality image Im1020, both of which have a resolution lower than a certain resolution set for the teacher data. In this case, the low-quality image Im1010 and the high-quality image Im1020 are each enlarged so as to have the certain resolution set for the teacher data. Furthermore, padding is performed on each of the enlarged low-quality image Im1010 and the high-quality image Im1020 so as to have the certain image size set for the teacher data. Then, the low-quality image Im1011 and the high-quality image Im1021 that have been enlarged and padded are paired, and the pair is used as one of the teacher data.

[0154] The certain image size set for the training data is the maximum possible image size when an image assumed to be processed (input image) is enlarged or reduced to a certain resolution. If the certain image size is not large enough, when an image input to the image quality improvement engine is enlarged, the image size may become too large for the machine learning model to handle.

[0155] In addition, the area where padding is performed is filled with a certain pixel value, filled with a neighboring pixel value, or mirror padded according to the characteristics of the machine learning model so that the image quality can be effectively improved. As in the first embodiment, an image having a specific shooting condition assumed to be a processing target is used as input data, and the specific shooting condition is a predetermined shooting part, shooting method, and shooting angle of view. In other words, unlike the first embodiment, the specific shooting condition according to this embodiment does not include the image size.

[0156] The image quality improvement unit 404 according to the present embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image to generate a high-image quality image. At this time, the image quality improvement unit 404 generates a deformed image by enlarging or reducing the input image so that it has a certain resolution set for the teacher data. The image quality improvement unit 404 also pads the deformed image so that it has a certain image size set for the teacher data to generate a padded image, and inputs the padded image to the image quality improvement engine.

[0157] Furthermore, the image quality improving unit 404 trims the high-image-quality padded image output from the image quality improving engine by the amount of the padded area, and generates a high-image-quality transformed image. Thereafter, the image quality improving unit 404 reduces or enlarges the generated high-image-quality transformed image so that it matches the image size of the input image, thereby generating a high-image-quality image.

[0158] Therefore, the image quality improving unit 404 according to this embodiment can improve the image quality of an input image having an image size that could not be handled in the first embodiment, and generate a high-image-quality image using the image quality improving engine.

[0159] Next, a series of image processing steps according to this embodiment will be described with reference to FIGS. 11 is a flow diagram of the image quality improvement process according to this embodiment. The processes of steps S510, S520, and S550 according to this embodiment are similar to those of the first embodiment, and therefore will not be described. When the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than the image size, the process of step S530 may be omitted and the process may proceed to step S540 after the process of step S520.

[0160] In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement possibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not determine the image size.

[0161] If the image quality improvement possibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

[0162] When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in Fig. 11 starts. In the image quality improvement process according to this embodiment, first, in step S1110, image quality improvement unit 404 enlarges or reduces the input image to a certain resolution set for the teacher data, and generates a deformed image.

[0163] Next, in step S1120, the image quality improvement unit 404 performs padding on the generated transformed image so that the image size becomes the image size set for the teacher data, and generates a padded image. At this time, the image quality improvement unit 404 performs padding with a fixed pixel value, padding with a neighboring pixel value, or mirror padding in accordance with the characteristics of the machine learning model so that the image quality of the area to be padded can be effectively improved.

[0164] In step S1130, the image quality improving unit 404 inputs the padding image to an image quality improving engine to obtain a high-image-quality padding image. Next, in step S1140, the image quality improving unit 404 performs trimming on the high-image-quality padding image by the area that was padded in step S1120, and generates a high-image-quality transformed image.

[0165] Thereafter, in step S1150, the image quality improvement unit 404 reduces or enlarges the high-quality transformed image to the image size of the input image to generate a high-quality image. When the image quality improvement unit 404 generates a high-quality image in step S1130, the image quality improvement process according to this embodiment ends, and the process proceeds to step S550. The process of step S550 is similar to step S550 in the first embodiment, and therefore a description thereof will be omitted.

[0166] As described above, the image quality improvement unit 404 according to this embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. Furthermore, the image quality improvement unit 404 generates a padded image by padding the input image whose image size has been adjusted so that the adjusted image size becomes an image size that can be handled by the image quality improvement engine, and inputs the padded image to the image quality improvement engine. After that, the image quality improvement unit 404 trims the output image from the image quality improvement engine by the amount of the padded area. Then, the image quality improvement unit 404 generates a high-quality image by adjusting the image size of the trimmed image to the original image size of the input image.

[0167] As a result, the image quality improvement unit 404 of this embodiment can generate a high-quality image by improving the image quality with the image quality improvement engine even for an input image of an image size that could not be handled in the first embodiment. Furthermore, by using an image quality improvement engine that has learned using teacher data based on the resolution, it may be possible to improve the image quality of an input image more efficiently than the image quality improvement engine of the fourth embodiment, which simply processes images of the same image size.

[0168] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0169] Sixth embodiment Next, an image processing device according to a sixth embodiment will be described with reference to Figures 4, 5, 12, and 13. In this embodiment, an image quality improving unit improves the image quality of an input image for each region of a certain image size, thereby generating a high-quality image.

[0170] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0171] The image quality improving unit 404 according to this embodiment is equipped with an image quality improving engine similar to that of the first embodiment. However, in this embodiment, the teacher data used for learning the image quality improving engine is different from the teacher data in the first embodiment. Specifically, pairs of input data, which is a low image quality image, and output data, which is a high image quality image, constituting the teacher data are constituted by rectangular area images of a certain image size in which the positional relationship in the low image quality image and the high image quality image corresponds. Note that the rectangular area is an example of a partial area, and does not have to be rectangular and may have any shape.

[0172] Here, the teacher data of the image quality improvement engine according to this embodiment will be described with reference to Fig. 12. As shown in Fig. 12, consider a case where one of the pairs constituting the teacher data includes, for example, an original image Im1210, which is a low image quality image, and a superimposed image Im1220, which is a high image quality image. In this case, in the first embodiment, the input data of the teacher data is Im1210, and the output data is Im1220.

[0173] In contrast, in this embodiment, the rectangular area image R1211 of the original image Im1210 is used as input data, and the rectangular area image R1221, which is the same shooting area as the rectangular area image R1211 in the superimposed image Im1220, is used as output data. Then, the rectangular area image R1211 as the input data and the rectangular area image R1221 as the output data form a pair of teacher data (hereinafter, a first rectangular area image pair). Here, the rectangular area image R1211 and the rectangular area image R1221 are images of a certain image size. Note that the original image Im1210 and the superimposed image Im1220 may be aligned by any method. Also, the corresponding positional relationship between the rectangular area image R1211 and the rectangular area image R1221 may be specified by any method such as template matching. Note that the image size and the number of dimensions of the input data and the output data may differ depending on the design of the image quality improvement engine. For example, when the processing target is an OCT image, if the input data is a part of a B-scan image (two-dimensional image), the output data may be a part of an A-scan image (one-dimensional image).

[0174] The fixed image size for the rectangular area images R1211 and R1221 can be determined, for example, from a common divisor of the pixel counts of the corresponding dimensions for a group of image sizes of an image assumed to be processed (input image). In this case, it is possible to prevent the positional relationship of the rectangular area images output by the image quality improvement engine from overlapping. Specifically, for example, consider a case where the image assumed to be processed is a two-dimensional image, and a first image size of the image size group is 500 pixels wide and 500 pixels high, and a second image size is 100 pixels wide and 100 pixels high. Here, a fixed image size for the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, the fixed image size is selected from 100 pixels wide and 100 pixels high, 50 pixels wide and 50 pixels high, 25 pixels wide and 25 pixels high, etc.

[0175] When the image to be processed is three-dimensional, the number of pixels is determined for the width, height, and depth. Note that a plurality of rectangular areas can be set for one pair of a low-quality image corresponding to the input data and a high-quality image corresponding to the output data. For this reason, for example, the rectangular area image R1212 of the original image Im1210 is set as input data, and the rectangular area image R1222, which is the same shooting area as the rectangular area image R1212 in the superimposed image Im1220, is set as output data. Then, a pair of teacher data is formed by the rectangular area image R1212, which is the input data, and the rectangular area image R1222, which is the output data. This makes it possible to create a rectangular area image pair other than the first rectangular area image pair.

[0176] In addition, by creating a large number of pairs of rectangular area images while changing the images of the rectangular area to images of different coordinates, the pairs constituting the teacher data can be enriched, and efficient image quality improvement can be expected by an image quality improvement engine that has learned using the teacher pairs. However, pairs that do not contribute to the image quality improvement of the machine learning model can be prevented from being added to the teacher data. For example, if a rectangular area image created from a high-quality image that is output data constituting a pair has image quality that is not suitable for diagnosis, the image output by the image quality improvement engine that has learned using such teacher data may also have image quality that is not suitable for image diagnosis. Therefore, pairs including such high-quality images can be removed from the teacher data.

[0177] In addition, for example, when the average luminance or luminance distribution of a pair of a rectangular area image created from a low-quality image and a rectangular area image created from a high-quality image differs greatly, such a pair can be removed from the training data. If learning is performed using such training data, the image quality improvement engine may output an image that is not suitable for image diagnosis because the image has a luminance distribution that differs greatly from that of the input image.

[0178] Furthermore, for example, consider a case where the structure or position of the object depicted in a pair of a rectangular area image created from a low-image quality image and a rectangular area image created from a high-image quality image are significantly different. In this case, there is a possibility that an image quality improvement engine that has learned using such training data will output an image that is not suitable for image diagnosis, in which the object depicted in the input image has a structure or position significantly different from that of the input image. Therefore, such pairs can be removed from the training data.

[0179] As in the first embodiment, the input data of the teacher data uses an image having a specific imaging condition assumed to be a processing target, and the specific imaging condition is a predetermined imaging part, imaging method, and imaging angle of view. That is, unlike the first embodiment, the specific imaging condition according to this embodiment does not include an image size.

[0180] The image quality improvement unit 404 according to the present embodiment uses an image quality improvement engine that has been trained using such teacher data to improve the image quality of an input image and generate a high-image quality image. At this time, the image quality improvement unit 404 divides the input image into a group of rectangular area images that are continuous without gaps and have a certain image size set for the teacher data. The image quality improvement unit 404 improves the image quality of each of the divided rectangular area images by using an image quality improvement engine to generate a group of high-image quality rectangular area images. After that, the image quality improvement unit 404 arranges and combines the generated high-image quality rectangular area images according to the positional relationship of the input image to generate a high-image quality image. Here, during learning, as long as the positional relationship between the input data and the output data, which are paired images, corresponds to each other, each rectangular area may be cut out (extracted) from any location in the low-image quality image and the high-image quality image. On the other hand, during image quality improvement, the input image may be divided into a group of rectangular area images that are continuous without gaps. In addition, the image size of each paired image during learning and the image size of each rectangular area image during image quality improvement may be set to correspond to each other (for example, to be the same). This makes it possible to improve learning efficiency while preventing problems such as unnecessary calculations or missing parts that result in an image not being generated.

[0181] In this manner, the image quality improvement unit 404 of this embodiment improves the image quality of the input image on a rectangular region basis, and by combining the improved image quality images, it is possible to improve the image quality of images of sizes that could not be handled in the first embodiment, thereby generating a high-image-quality image.

[0182] Next, a series of image processing according to this embodiment will be described with reference to Figs. 5, 13, and 14. Fig. 13 is a flow diagram of image quality improvement processing according to this embodiment. Note that the processing of steps S510, S520, and S550 according to this embodiment is similar to these processing in the first embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to shooting conditions other than the image size, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0183] In step S520, as in the first embodiment, when the imaging condition acquisition unit 402 acquires a group of imaging conditions for the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the acquired group of imaging conditions to determine whether the image quality improvement engine provided in the image quality improvement unit 404 can handle the input image. Specifically, the image quality improvement possibility determination unit 403 determines whether the imaging conditions of the input image are imaging body part, imaging method, and imaging angle of view that the image quality improvement engine can handle. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not determine the image size.

[0184] If the image quality improvement possibility determination unit 403 determines the imaging part, imaging method, and imaging angle of view, and determines that the input image can be handled, the process proceeds to step S540. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the settings and implementation form of the image processing device 400, the image quality improvement process in step S540 may be performed even if it is determined that the input image cannot be processed based on some of the imaging part, imaging method, and imaging angle of view.

[0185] When the process proceeds to step S540, the image quality improvement process according to the present embodiment shown in FIG. 13 is started. This will be described with reference to FIG. 14. In the image quality improvement process according to the present embodiment, first, in step S1310, as shown in FIG. 14(a), an input image is divided into a group of rectangular area images of a certain image size (size shown in R1411) set for the teacher data, which are continuous without gaps. Here, FIG. 14(a) shows an example in which an input image Im1410 is divided into a group of rectangular area images R1411 to R1426 of a certain image size. As described above, depending on the design of the image quality improvement engine, the image sizes and the number of dimensions of the input image and the output image of the image quality improvement engine may differ. In this case, the division positions of the input image can be adjusted by overlapping or separating them so that the combined high-image-quality image generated in step S1320 has no defects. FIG. 14(b) shows an example in which the division positions are overlapped. In FIG. 14(b), R1411' and R1412' show overlapping areas. Although not shown to avoid complication, R1413 to R1426 also have similar overlapping areas R1413' to R1426'. The rectangular area size set for the teacher data in the case of FIG. 14(b) is the size shown in R1411'. Since there is no data in the periphery (top, bottom, left, right, and bottom) of the outside of the input image Im1410, the area is filled with a certain pixel value, filled with a neighboring pixel value, or mirror padded. In addition, depending on the image quality improvement engine, the accuracy of image quality improvement may decrease in the periphery (top, bottom, left, and right) of the inside of the image due to filter processing. Therefore, as shown in FIG. 14(b), rectangular area images may be set by overlapping the division positions, and a part of the rectangular area images may be trimmed and synthesized as the final image. The size of the rectangular area is set according to the characteristics of the image quality improvement engine. Although OCT tomographic images are shown in Fig. 14(a) and (b), the input image (Im1450) may be a front image such as an OCTA En-Face image, as shown in Fig. 14(c) and (d), and similar processing is possible. The size of the rectangular area image is set appropriately depending on the target image and the type of image quality improvement engine.

[0186] Next, in step S1320, the image quality improvement unit 404 improves the image quality of each of the rectangular area images R1411 to R1426, or, if an overlapping area is set, the rectangular area images R1411' to R1426', using an image quality improvement engine to generate a high-image-quality rectangular area image group.

[0187] Then, in step S1330, the image quality improving unit 404 arranges each of the generated high-image-quality rectangular area images in the same positional relationship as each of the rectangular area images R1411 to R1426 divided from the input image, and combines them to generate a high-image-quality image. When an overlapping area is set, the rectangular area images R1411' to R1426' are arranged in the same positional relationship as each of them, and then the rectangular area images R1411 to R1426 are cut out and combined to generate a high-image-quality image. Note that the overlapping area may be used to correct the luminance values ​​of the rectangular area images R1411' to R1426'. For example, a rectangular area image to be used as a reference is arbitrarily set. Then, the luminance values ​​of the same coordinate points are measured in adjacent rectangular images having an overlapping area with the reference rectangular image, and the difference (ratio) in luminance values ​​between the adjacent images can be found. Similarly, by finding the difference (ratio) in luminance values ​​in the overlapping areas in all images, correction can be performed to eliminate unevenness in luminance values ​​as a whole. It is not necessary to use the entire overlapping area for brightness value correction, and a part of the overlapping area (several peripheral pixels) may not be used.

[0188] As described above, the image quality improving unit 404 according to this embodiment divides an input image into a plurality of rectangular area images (third images) R1411-R1426 of a predetermined image size. After that, the image quality improving unit 404 inputs the divided rectangular area images R1411-R1426 to an image quality improving engine to generate a plurality of fourth images, and integrates the plurality of fourth images to generate a high image quality image. Note that if rectangular area groups overlap in positional relationship during integration, the pixel values ​​of the rectangular area groups can be integrated or overwritten.

[0189] As a result, the image quality improving unit 404 of this embodiment can improve the image quality of an input image of an image size that could not be handled in the first embodiment, and generate a high-image-quality image by using an image quality improving engine. Also, if the teacher data is created from a plurality of images obtained by dividing low-image quality images and high-image quality images into a predetermined image size, a large amount of teacher data can be created from a small number of images. Therefore, in this case, the number of low-image quality images and high-image quality images for creating the teacher data can be reduced.

[0190] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0191] Seventh embodiment Next, an image processing device according to a seventh embodiment will be described with reference to Figs. 15 to 17. In this embodiment, an image quality assessment unit selects the image with the highest image quality from among multiple high-image-quality images output from multiple image quality improvement engines in response to an instruction from an examiner.

[0192] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0193] Fig. 15 shows a schematic configuration of an image processing device 1500 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image quality evaluation unit 1506 is provided in the image processing device 1500 according to this embodiment. Note that the image processing device 1500 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in Fig. 4, and a description thereof will be omitted.

[0194] Furthermore, the image processing device 1500 may be connected to the photographing device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

[0195] The image quality improving unit 404 according to this embodiment is equipped with two or more image quality improving engines that have been subjected to machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. Specifically, first, a pair group of input data, which is a low-quality image, and output data, which is a high-quality image, captured under various shooting conditions is prepared. Next, the pair group is grouped according to an arbitrary combination of shooting conditions to create a teacher data group. For example, a first teacher data group is created that is composed of a pair group acquired under a first combination of shooting conditions, and a second teacher data group is created that is composed of a pair group acquired under a second combination of shooting conditions.

[0196] Then, machine learning is performed in separate image quality improvement engines using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data, and a second image quality improvement engine corresponding to a machine learning model trained with the first training data.

[0197] Since each of these image quality improvement engines uses different teacher data for training the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for an input image captured and acquired under a first combination of shooting conditions, and a low degree of image quality improvement for an image captured and acquired under a second combination of shooting conditions. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for an input image captured and acquired under a second combination of shooting conditions, and a low degree of image quality improvement for an input image captured and acquired under a first shooting condition.

[0198] Since each of the teacher data is composed of a pair group grouped by a combination of imaging conditions, the image quality tendencies of the image groups constituting the pair groups are similar. Therefore, the image quality improvement engine can improve image quality more effectively than the image quality improvement engine according to the first embodiment, as long as the combination of the corresponding imaging conditions is used. The combination of imaging conditions for grouping the pairs of teacher data may be any combination, and may be, for example, a combination of two or more of the imaging site, imaging angle of view, and image resolution. Furthermore, the grouping of teacher data may be performed based on one imaging condition, as in the second embodiment.

[0199] An image quality assessment unit 1506 selects the high-quality image with the highest image quality from the multiple high-quality images generated by the image quality improvement unit 404 using multiple image quality improvement engines, in response to an instruction from the examiner.

[0200] The output unit 405 can display the high-quality image selected by the image quality assessment unit 1506 on the display unit 20 or output it to another device. The output unit 405 can display a plurality of high-quality images generated by the image quality improvement unit 404 on the display unit 20, and the image quality assessment unit 1506 can select the high-quality image with the highest image quality in response to an instruction from the examiner who checks the display unit 20.

[0201] This allows the image processing device 1500 to output the high-quality image with the highest image quality in accordance with the examiner's instructions from among a plurality of high-quality images generated using a plurality of image quality improvement engines.

[0202] A series of image processing according to this embodiment will be described below with reference to Figs. 16 and 17. Fig. 16 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S1610 and S1620 according to this embodiment is similar to the processing of steps S510 and S520 in the first embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S1620, the processing of step S1630 may be omitted and the processing may proceed to step S1640.

[0203] In step S1620, as in the first embodiment, when the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S1630. In step S1630, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image, as in the second embodiment.

[0204] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed, as in the first embodiment, even if it is determined that some shooting conditions cannot be handled by the image quality improvement engines.

[0205] In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engines, and generates a group of high image quality images.

[0206] In step S1650, the image quality assessment unit 1506 selects the image with the highest image quality from the group of high-image-quality images generated in step S1640. Specifically, first, the output unit 405 causes the group of high-image-quality images generated in step S1640 to be displayed on the user interface of the display unit 20.

[0207] An example of the interface is shown in Fig. 17. The interface displays an input image Im1710 and high-quality images Im1720, Im1730, Im1740, and Im1750 output by each of the image quality improvement engines. The examiner operates an arbitrary input device (not shown) to specify the image with the highest image quality, that is, the image most suitable for image diagnosis, from the image group (high-quality images Im1720 to Im1750). Note that since an input image that has not been improved in image quality by the image quality improvement engine may be more suitable for image diagnosis, the input image may be added to the image group that is the subject of the examiner's specification.

[0208] Thereafter, the image quality assessment unit 1506 selects the high-quality image designated by the examiner as the image with the highest quality.

[0209] In step S1660, the output unit 405 displays the image selected in step S1650 on the display unit 20 or outputs it to another device. However, if it is determined in step S1630 that the input image cannot be processed, the output unit 405 outputs the input image as an output image. Note that, if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0210] As described above, the image quality improving unit 404 according to this embodiment generates a plurality of high image quality images from an input image using a plurality of image quality improving engines, and the output unit 405 of the image processing device 1500 outputs at least one image of the plurality of high image quality images in response to an instruction from the examiner. In particular, in this embodiment, the output unit 405 outputs the image with the highest image quality in response to an instruction from the examiner. This allows the image processing device 1500 to output a high-quality image having high image quality in accordance with an instruction from the examiner, from among a plurality of high-quality images generated using a plurality of image quality improvement engines.

[0211] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0212] <Eighth embodiment> Next, an image processing device according to an eighth embodiment will be described with reference to Figs. 15 and 16. In this embodiment, an image quality evaluation unit uses an image quality evaluation engine to select an image with the highest image quality from among multiple high-image quality images output from multiple image quality improvement engines.

[0213] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 1500 according to the seventh embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the seventh embodiment. Since the configuration of the image processing device according to this embodiment is the same as that of the image processing device according to the seventh embodiment, the configuration shown in FIG. 15 is indicated by the same reference numerals, and description thereof will be omitted.

[0214] The image quality evaluation unit 1506 according to this embodiment is provided with an image quality evaluation engine that evaluates the image quality of an input image. The image quality evaluation engine outputs an image quality evaluation index for the input image. The image quality evaluation processing method for calculating the image quality evaluation index in the image quality evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm. The pair of input data constituting the teacher data for training the machine learning model is an image group consisting of a low image quality image group and a high image quality image group captured in advance under various shooting conditions. In addition, the pair of output data constituting the teacher data for training the machine learning model is, for example, a group of image quality evaluation indices set for each of the image groups of the input data by an examiner performing image diagnosis.

[0215] Next, a series of image processing according to this embodiment will be described with reference to Fig. 16. Note that the processing of steps S1610, S1620, S1630, and S1660 according to this embodiment is similar to those in the seventh embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S1620, the processing of step S1630 may be omitted and the processing may proceed to step S1640.

[0216] In step S1630, similarly to the seventh embodiment, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engines can handle the input image, the process proceeds to step S1640. Note that, similarly to the first embodiment, depending on the settings and implementation form of the image processing device 400, step S1640 may be performed even if it is determined that some shooting conditions cannot be handled by the image quality improvement engine.

[0217] In step S1640, the image quality improving unit 404 inputs the input image acquired in step S1610 to each of the image quality improving engines, and generates a group of high image quality images.

[0218] In step S1650, the image quality evaluation unit 1506 selects the image with the highest image quality from the group of high-image-quality images generated in step S1640. Specifically, first, the image quality evaluation unit 1506 inputs the group of high-image-quality images generated in step S1640 to an image quality evaluation engine. The image quality evaluation engine calculates an image quality evaluation index for each input high-image-quality image based on learning. The image quality evaluation unit 1506 selects the high-image-quality image with the highest image quality evaluation index calculated from the calculated image quality evaluation indexes. Note that since an input image that has not been improved in image quality by the image improvement engine may be more suitable for image diagnosis, the image quality evaluation unit 1506 may also input the input image to the image quality evaluation engine and add the image quality evaluation index for the input image to the selection. Step S1660 is similar to step S1660 in the seventh embodiment, and therefore a description thereof will be omitted.

[0219] As described above, the image processing device 1500 according to this embodiment further includes an image quality evaluation unit 1506 that evaluates the image quality of a high-quality image. The image quality improvement unit 404 generates a plurality of high-quality images from an input image using a plurality of image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the plurality of high-quality images according to the evaluation result by the image quality evaluation unit 1506. In particular, the image quality evaluation unit 1506 according to this embodiment includes an image quality evaluation engine that uses an evaluation value by a predetermined evaluation method as learning data. The image quality evaluation unit 1506 selects the high-quality image that has the highest evaluation result by the image quality evaluation unit 1506 using the image quality evaluation engine from the plurality of high-quality images. The output unit 405 outputs the high-quality image with the highest evaluation value selected by the image quality evaluation unit 1506.

[0220] As a result, the image processing device 1500 according to this embodiment can easily output the high-quality image most suitable for image diagnosis from among a plurality of high-quality images, based on the output of the image quality assessment engine.

[0221] In this embodiment, the image quality evaluation unit 1506 selects a high-quality image having the highest image quality evaluation index among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 displays the selected high-quality image on the display unit 20. However, the configuration of the image quality evaluation unit 1506 is not limited to this. For example, the image quality evaluation unit 1506 may select high-quality images having several image quality evaluation indices that are among the image quality evaluation indices output by the image quality evaluation engine, and the output unit 405 may display the selected high-quality image on the display unit 20. Furthermore, the output unit 405 may display the image quality evaluation indices output by the image quality evaluation engine together with the corresponding high-quality image on the display unit 20, and the image quality evaluation unit 1506 may select the image with the highest image quality in accordance with an instruction from the examiner.

[0222] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1500, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0223] <Ninth embodiment> Next, an image processing device according to a ninth embodiment will be described with reference to Figs. 18 and 19. In this embodiment, an authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether or not the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

[0224] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0225] 18 shows a schematic configuration of an image processing device 1800 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 1800 according to this embodiment is provided with an authenticity evaluation unit 1807. The image processing device 1800 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the same reference numerals are used for the configuration shown in FIG. 4, and a description thereof will be omitted.

[0226] Furthermore, the image processing device 1800 may be connected to the photographing device 10, the display unit 20, and other devices (not shown) via any circuit or network, similar to the image processing device 400 according to the first embodiment. Furthermore, these devices may be connected to any other device via a circuit or network, or may be configured integrally with any other device. Although these devices are separate devices in this embodiment, some or all of these devices may be configured integrally.

[0227] The authenticity evaluation unit 1807 is provided with an authenticity evaluation engine. The authenticity evaluation unit 1807 uses the authenticity evaluation engine to evaluate whether or not the high-quality image generated by the image quality improvement engine has been sufficiently improved in quality. The authenticity evaluation processing method in the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

[0228] The teacher data for training the machine learning model includes a pair group of high-quality images taken under various shooting conditions in advance and a label (hereinafter, genuine label) indicating that the images were taken and acquired by the target shooting device. The teacher data also includes a pair group of high-quality images generated by inputting low-quality images to a high-quality image enhancement engine with low image enhancement accuracy and a label (hereinafter, counterfeit label) indicating that the images were not taken and acquired by the target shooting device.

[0229] An authenticity evaluation engine that has been trained using such teacher data cannot reliably evaluate whether an input image is an image that has been photographed and acquired by an imaging device, but it can evaluate whether the image has the resemblance of an image that has been photographed and acquired by an imaging device. By utilizing this characteristic, the authenticity evaluation unit 1807 can input a high-quality image generated by the image quality improvement unit 404 to the authenticity evaluation engine, and evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality.

[0230] When the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality, the output unit 405 displays the high-quality image on the display unit 20. On the other hand, when the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved in quality, the output unit 405 displays the input image on the display unit 20. Note that, when displaying the input image, the output unit 405 can display on the display unit 20 that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved in quality or that the image being displayed is the input image.

[0231] A series of image processing according to this embodiment will be described below with reference to Fig. 19. Fig. 19 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S1910 to S1940 according to this embodiment is similar to the processing of steps S510 to S540 in the first embodiment, and therefore a description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S1920, the processing of step S1930 may be omitted and the processing may proceed to step S1940.

[0232] In step S1940, when the image quality improving unit 404 generates a group of high-image-quality images, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-image-quality images generated in step S1940 to an authenticity evaluation engine, and performs an authenticity evaluation based on the output of the authenticity evaluation engine. Specifically, when an authenticity evaluation engine outputs an authenticity label (authentic), the authenticity evaluation unit 1807 evaluates that the generated high-image-quality images have been sufficiently improved in image quality. On the other hand, when an authenticity evaluation engine outputs a counterfeit label (fake), the authenticity evaluation unit 1807 evaluates that the generated high-image-quality images have not been sufficiently improved in image quality.

[0233] In step S1960, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in quality, the output unit 405 causes the high-quality image to be displayed on the display unit 20. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not been sufficiently improved in quality, the output unit 405 causes the display unit 20 to display the input image.

[0234] As described above, the image processing device 1800 according to this embodiment further includes an authenticity evaluation unit 1807 that evaluates the image quality of a high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of an image. The authenticity evaluation engine includes a machine learning engine that uses as learning data an image generated by an image quality improvement engine that has lower (worse) accuracy of image quality improvement processing than the image quality improvement engine of the image quality improvement unit 404. The output unit 405 of the image processing device 1800 outputs the high-quality image when the output from the authenticity evaluation engine of the authenticity evaluation unit is true.

[0235] As a result, in the image processing device 1800 according to this embodiment, the examiner can efficiently check a high-quality image that has been sufficiently improved in quality.

[0236] In addition, the machine learning model of the image quality improvement engine and the machine learning model of the authenticity evaluation engine may be trained in cooperation with each other to improve the efficiency and accuracy of both engines.

[0237] In this embodiment, the image quality improvement unit 404 generates one high-quality image, and the authenticity evaluation unit 1807 evaluates the generated high-quality image, but the evaluation by the authenticity evaluation unit 1807 is not limited to this. For example, as in the second embodiment, when the image quality improvement unit 404 generates multiple high-quality images using multiple image quality improvement engines, the authenticity evaluation unit 1807 may evaluate at least one of the multiple high-quality images generated. In this case, for example, the authenticity evaluation unit 1807 may evaluate all of the multiple high-quality images generated, or may evaluate only the image specified by the examiner among the multiple high-quality images.

[0238] Furthermore, the output unit 405 may cause the display unit 20 to display the result of the judgment made by the authenticity evaluation unit 1807 as to whether or not the high-quality image has been sufficiently improved in quality, and output the high-quality image in response to the examiner's instructions.

[0239] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 1800, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0240] <Tenth embodiment> Next, an image processing device according to a tenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, inputs the images to an image quality improvement engine, and combines output images from the image quality improvement engine to generate a three-dimensional high-quality image.

[0241] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0242] The acquisition unit 401 according to the present embodiment acquires a three-dimensional image composed of a group of structurally continuous two-dimensional images. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a group of OCT B-scan images (tomographic images). Also, for example, the three-dimensional image is a three-dimensional CT volume image composed of a group of axial tomographic images.

[0243] The image quality improvement unit 404 is provided with an image quality improvement engine, as in the first embodiment. Note that a group of pairs of input data and output data, which are teacher data for the image quality improvement engine, are composed of a group of two-dimensional images. The image quality improvement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, and inputs each of the two-dimensional images to the image quality improvement engine. This allows the image quality improvement unit 404 to generate a plurality of two-dimensional high-quality images.

[0244] The output unit 405 combines the multiple two-dimensional high quality images generated by the image quality improving unit 404 for each of the two-dimensional images of the three-dimensional image, and outputs a three-dimensional high quality image.

[0245] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5. Note that the processing of steps S510 to S530 and step S550 according to this embodiment is similar to those in the first embodiment, and therefore the description will be omitted. However, in step S510, the acquisition unit 401 acquires a three-dimensional image. Note that, when the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0246] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be handled by the image quality improvement engine, the process proceeds to step S540. The image quality improvement possibility determination unit 403 may perform the determination based on the shooting conditions of the three-dimensional image, or may perform the determination based on the shooting conditions of the multiple two-dimensional images that constitute the three-dimensional image. In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into multiple two-dimensional images. The image quality improvement unit 404 inputs each of the multiple divided two-dimensional images to the image quality improvement engine to generate multiple two-dimensional high-quality images. The image quality improvement unit 404 combines the multiple generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

[0247] In step S550, the output unit 405 causes the generated high quality three-dimensional image to be displayed on the display unit 20. Note that the high quality three-dimensional image may be displayed in any manner.

[0248] As described above, the image quality improvement unit 404 according to this embodiment divides a three-dimensional input image into multiple two-dimensional images and inputs them to the image quality improvement engine. The image quality improvement unit 404 combines the multiple two-dimensional high-quality images output from the image quality improvement engine to generate a three-dimensional high-quality image.

[0249] As a result, the image quality improving unit 404 according to this embodiment can improve the image quality of a three-dimensional image by using an image quality improving engine that has been trained using teacher data of a two-dimensional image.

[0250] Note that the output unit 405 may output the generated high-quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0251] <Eleventh embodiment> Next, an image processing device according to an eleventh embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improvement unit divides a three-dimensional input image into a plurality of two-dimensional images, improves the image quality of the plurality of two-dimensional images in parallel by a plurality of image quality improvement engines, and generates a high-quality three-dimensional image by combining output images from the image quality improvement engines.

[0252] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the tenth embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing device according to the tenth embodiment. Note that since the configuration of the image processing device according to this embodiment is the same as those of the image processing devices according to the first and tenth embodiments, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0253] The image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines similar to those in the tenth embodiment. Note that the plurality of image quality improvement engines included in the image quality improvement unit 404 may be implemented in two or more device groups via a circuit or a network so as to enable distributed processing, or may be implemented in a single device.

[0254] The image quality improving unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images, as in the tenth embodiment. The image quality improving unit 404 uses a plurality of image quality improving engines to improve the image quality of the plurality of two-dimensional images in a shared manner (in parallel), and generates a plurality of two-dimensional high-quality images. The image quality improving unit 404 combines a plurality of two-dimensional high-quality images output from a plurality of image quality improving engines based on the three-dimensional image to be processed, and generates a three-dimensional high-quality image.

[0255] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5. Note that the processing of steps S510 to S530 and step S550 according to this embodiment is similar to those in the tenth embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0256] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be processed by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may perform the determination based on the shooting conditions of the three-dimensional image, or may perform the determination based on the shooting conditions of the multiple two-dimensional images that constitute the three-dimensional image.

[0257] In step S540, the image quality improvement unit 404 divides the acquired three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided two-dimensional images into a plurality of image quality improvement engines, and performs image quality improvement processing in parallel to generate a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated two-dimensional high-quality images based on the acquired three-dimensional image to generate a three-dimensional high-quality image.

[0258] In step S550, the output unit 405 causes the generated high quality three-dimensional image to be displayed on the display unit 20. Note that the high quality three-dimensional image may be displayed in any manner.

[0259] As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines. The image quality improvement unit 404 divides a three-dimensional input image into a plurality of two-dimensional images, and generates a plurality of two-dimensional high-quality images using the plurality of image quality improvement engines in parallel. The image quality improvement unit 404 generates a three-dimensional high-quality image by integrating the plurality of two-dimensional high-quality images.

[0260] As a result, the image quality improving unit 404 according to this embodiment can improve the image quality of a 3D image by using an image quality improving engine that has been trained using teacher data of a 2D image. Also, compared to the tenth embodiment, the image quality of a 3D image can be improved more efficiently.

[0261] The teacher data for the multiple image quality improvement engines may be different teacher data depending on the processing target to be processed by each image quality improvement engine. For example, the first image quality improvement engine may learn from teacher data for the first imaging region, and the second image quality improvement engine may learn from teacher data for the second imaging region. In this case, each image quality improvement engine can improve the image quality of the two-dimensional image with higher accuracy.

[0262] Also, the output unit 405 may output the generated high quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high performance imaging device, high cost processing, and noise reduction processing may be used.

[0263] <Twelfth embodiment> Next, an image processing apparatus according to a twelfth embodiment will be described with reference to Figs. 5 and 20. In this embodiment, an acquisition unit 401 acquires an input image from an image management system 2000, not from an image capture device.

[0264] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device 400 according to the first embodiment, the same reference numerals are used for the configuration shown in FIG. 4 and the description thereof will be omitted.

[0265] 20 shows a schematic configuration of an image processing device 400 according to this embodiment. The image processing device 400 according to this embodiment is connected to an image management system 2000 and a display unit 20 via any circuit or network. The image management system 2000 is a device and system that receives and stores images captured by any imaging device or images that have been subjected to image processing. In addition, the image management system 2000 can transmit images in response to a request from a connected device, perform image processing on stored images, and request image processing from other devices. The image management system can include, for example, a picture archiving and communication system (PACS).

[0266] The acquisition unit 401 according to this embodiment can acquire an input image from an image management system 2000 connected to the image processing device 400. In addition, the output unit 405 can output a high-quality image generated by the image quality improvement unit 404 to the image management system 2000.

[0267] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5. Note that the processing of steps S520 to S540 according to this embodiment is similar to those of the first embodiment, and therefore the description will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0268] In step S510, the acquisition unit 401 acquires an image stored in the image management system 2000 as an input image from the image management system 2000 connected via a circuit or a network. The acquisition unit 401 may acquire the input image in response to a request from the image management system 2000. Such a request may be issued, for example, when the image management system 2000 stores an image, before transmitting the stored image to another device, or when displaying the stored image on the display unit 20. The request may also be issued, for example, when a user operates the image management system 2000 to request image quality improvement processing, or when a high-quality image is used for an image analysis function provided in the image management system 2000.

[0269] The processing of steps S520 to S540 is the same as that in the first embodiment. When image quality improving section 404 generates a high-image-quality image in step S540, the processing proceeds to step S550. In step S550, if a high-image-quality image has been generated in step S540, output section 405 outputs the high-image-quality image to image management system 2000 as an output image. If a high-image-quality image has not been generated in step S540, output section 405 outputs the input image to image management system 2000 as an output image. Depending on the settings and implementation of image processing device 400, output section 405 may process the output image so that it can be used by image management system 2000, or convert the data format of the output image.

[0270] As described above, the acquisition unit 401 according to this embodiment acquires an input image from the image management system 2000. Therefore, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis based on an image stored in the image management system 2000, without increasing the invasiveness or labor of the photographer or the subject. The output high-quality image can be stored in the image management system 2000 or displayed on a user interface provided in the image management system 2000. The output high-quality image can be used for an image analysis function provided in the image management system 2000, or can be transmitted to another device connected to the image management system 2000 via the image management system 2000.

[0271] The image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via a circuit or a network. Although these devices are separate devices in this embodiment, some or all of these devices may be integrated into one device.

[0272] Furthermore, the output unit 405 may output the generated high-quality image to the image management system 2000 or another device connected to the image processing device 400, as in the first embodiment.

[0273] <Thirteenth embodiment> Next, an image processing device according to a thirteenth embodiment will be described with reference to Figures 4, 5, 21A, and 21B. In this embodiment, an image quality improving unit receives a plurality of images as input images and generates one high-quality image.

[0274] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0275] The acquisition unit 401 according to this embodiment acquires a plurality of images as input data to be processed from the image capture device 10 or another device.

[0276] The image quality improving unit 404 according to this embodiment is provided with an image quality improving engine similar to that of the first embodiment. The teacher data may also be similar to that of the first embodiment. The image quality improving unit 404 inputs each of the multiple images acquired by the acquisition unit 401 to the image quality improving engine, and performs a process of overlaying the multiple high-quality images outputted to generate a final high-quality image. Note that the image quality improving section 404 may align the multiple high-image-quality images by any method before overlaying the multiple high-image-quality images.

[0277] The output unit 405 displays the final high-quality image generated by the image quality improving unit 404 on the display unit 20. The output unit 405 may display a plurality of input images on the display unit 20 together with the final high-quality image. The output unit 405 may also display the generated plurality of high-quality images on the display unit 20 together with the final high-quality image and the input image.

[0278] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5 and Fig. 21A. Fig. 21A is a flow diagram of image quality improvement processing according to this embodiment. Note that the processing of steps S510 to S530 according to this embodiment is similar to those in the first embodiment, and therefore description thereof will be omitted.

[0279] However, in step S510, the acquisition unit 401 acquires a plurality of images, and in steps S520 and S530, the shooting conditions are acquired for each of the plurality of images, and it is determined whether or not the image quality can be improved by the image quality improvement engine. If the image quality of the input image is improved unconditionally with respect to the shooting conditions, the process of step S530 may be omitted after the process of step S520, and the process may proceed to step S540. Also, if it is determined that some of the images cannot be handled by the image quality improvement engine, the image may be excluded from the subsequent processes.

[0280] In step S530, if the image quality improvement possibility determination unit 403 determines that the multiple input images can be handled by the image quality improvement engine, the process proceeds to step S540. When the process proceeds to step S540, the image quality improvement process according to this embodiment shown in Fig. 21A is started. In the image quality improvement process according to this embodiment, first, in step S2110, the image quality improvement unit 404 inputs each of the multiple input images to the image quality improvement engine and generates a group of high image quality images.

[0281] Next, in step S2120, the image quality improvement unit 404 performs a process of superimposing the generated high-quality images to generate a single final high-quality image. The superimposing process may be performed by an averaging process such as an arithmetic mean or any other existing process. When superimposing, the image quality improvement unit 404 may superimpose the multiple high-quality images after aligning them by any method. Once the image quality improvement unit 404 has generated the final high-quality image, the process proceeds to step S550.

[0282] In step S550, the output unit 405 causes the display unit 20 to display the generated final high quality image.

[0283] As described above, the image quality improvement unit 404 according to this embodiment generates one final high-quality image from multiple input images. Since the image quality improvement by the image quality improvement engine is based on the input image, for example, if a lesion or the like is not displayed properly in a certain input image, the pixel value will be low in the high-quality image obtained by improving the image quality of the input image. On the other hand, there are cases where the lesion or the like is displayed properly in another input image obtained by photographing the same location, and the pixel value is high in the high-quality image obtained by improving the image quality of the other input image. Therefore, by superimposing these high-quality images, the low or high pixel value can be displayed properly, and a high-quality image with high contrast can be generated. In addition, by making the number of input images smaller than the number required for conventional superimposition, the cost of the conventional increase in imaging time, etc. can be reduced.

[0284] This effect becomes more pronounced when using input images that use motion contrast data such as OCTA.

[0285] Since the motion contrast data is obtained by detecting the temporal change of the object during the time interval when the same part of the object is repeatedly photographed, for example, only a slight movement of the object may be detected during a certain time interval. On the other hand, when photographing is performed during a different time interval, the movement of the object may be detected as a larger movement. Therefore, by superimposing the motion contrast images of each case with high image quality, it is possible to interpolate the motion contrast that did not occur or was only slightly detected at a specific timing. Therefore, according to such processing, a motion contrast image in which the contrast is enhanced for more movements of the object can be generated, and the examiner can grasp the state of the object more accurately.

[0286] Therefore, when an image depicting an area that changes over time, such as an OCTA image, is used as the input image, a specific area of ​​the subject can be imaged in more detail by overlaying high-quality images acquired at different times.

[0287] In this embodiment, a high quality image is generated from each of a plurality of input images, and the high quality images are superimposed to generate a final high quality image, but the method of generating a high quality image from a plurality of input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in Fig. 21B, when the image quality improvement process is started in step S540, in step S2130, the image quality improvement unit 404 superimposes the input images to generate a single superimposed input image.

[0288] Then, in step S2140, the image quality improvement unit 404 inputs one superimposed input image to an image quality improvement engine to generate one high-quality image. Even with this kind of image quality improvement process, as with the above-mentioned image quality improvement process, it becomes possible to appropriately display parts of multiple input images that have low or high pixel values, and to generate a high-quality image with high contrast. This process can also have a remarkable effect when a motion contrast image such as the above-mentioned OCTA image is used as the input image.

[0289] When performing the image quality improvement process, the image quality improvement engine uses superimposed images of the same number of input images as the multiple input images to be processed as input data for the teacher data of the image quality improvement engine, thereby enabling the image quality improvement engine to perform appropriate image quality improvement processing.

[0290] In addition, in the image quality improvement process according to the present embodiment and the other image quality improvement processes described above, the process of combining the high image quality image group or the input image group is not limited to superimposition. For example, a single image may be generated by applying a MAP estimation process to these image groups. Also, a single image may be generated by synthesizing the high image quality image group or the input image group.

[0291] In the case of generating a single image by synthesizing a group of high-quality images or a group of input images, for example, an image having a wide gradation in the high-luminance region and an image having a wide gradation in the low-luminance region may be used as the input image. In this case, for example, an image obtained by improving the image quality of an image having a wide gradation in the high-luminance region and an image obtained by improving the image quality of an image having a wide gradation in the low-luminance region are synthesized. This makes it possible to generate an image that can express a wider range of brightness (dynamic range). In this case, the input data of the teacher data of the high-quality engine can be an image having a wide gradation in the high-luminance region or a low-quality image having a wide gradation in the low-luminance region, which are to be processed. In addition, the output data of the teacher data of the high-quality engine can be a high-quality image corresponding to the input data.

[0292] Also, an image having a wide range of gradations in the high brightness region and an image having a wide range of gradations in the low brightness region may be synthesized, and the synthesized image may be enhanced in image quality by the image quality enhancement engine. In this case, an image capable of expressing a wider range of brightness may be generated. In this case, the input data of the teacher data of the image quality enhancement engine may be an image obtained by synthesizing a low quality image having a wide range of gradations in the high brightness region and a low quality image having a wide range of gradations in the low brightness region, which are to be processed. Also, the output data of the teacher data of the image quality enhancement engine may be a high quality image corresponding to the input data.

[0293] In these cases, using a high image quality engine, it is possible to improve the image quality of images that can express a wider range of brightness, making it possible to process fewer images than before, and providing images suitable for image analysis at a lower cost.

[0294] In addition, as a method of photographing an image having a wide gradation in the high luminance region and an image having a wide gradation in the low luminance region, any method may be adopted, such as shortening or lengthening the exposure time of the photographing device. In addition, the method of dividing the gradation width is not limited to the low luminance region and the high luminance region, and may be any method.

[0295] In addition, in the image quality improvement process according to this embodiment, a plurality of input images may be processed in parallel using a plurality of image quality improvement engines. The output unit 405 may output the generated high-quality image to the image capture device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high-quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing may be used.

[0296] <Fourteenth embodiment> Next, an image processing device according to a fourteenth embodiment will be described with reference to Figs. 4 and 5. In this embodiment, an image quality improving unit uses a medium quality image generated from a plurality of low quality images as an input image and generates a high quality image.

[0297] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0298] The acquisition unit 401 according to this embodiment acquires a medium quality image obtained by superimposing a plurality of low quality images as input data to be processed from the photographing device 10 or another device. When superimposing the low quality images, any position alignment process may be performed.

[0299] The image quality improvement unit 404 according to this embodiment is equipped with an image quality improvement engine similar to that of the first embodiment. However, the image quality improvement engine of this embodiment is designed to input a medium image quality image, which is of medium image quality, and output a high image quality image. A medium image quality image is a composite image generated by superimposing a group of low image quality images. A high image quality image is an image of higher image quality than a medium image quality image. Also, for the pair groups constituting the teacher data used in training the image quality improvement engine, the input data constituting each pair is a medium image quality image generated in the same manner as the medium image quality image, and the output data is a high image quality image.

[0300] The output unit 405 causes the display unit 20 to display the high quality image generated by the image quality improving unit 404 . In addition, the output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from multiple low-quality images.

[0301] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that since the processing steps S520 to S550 according to this embodiment are similar to those steps in the first embodiment, the description thereof will be omitted.

[0302] In step S510, the acquisition unit 401 acquires a medium quality image as an input image from the photographing device 10 or another device. The acquisition unit 401 may acquire a medium quality image generated by the photographing device 10 as an input image in response to a request from the photographing device 10. Such a request may be issued, for example, when the photographing device 10 generates an image, before or after the image generated by the photographing device 10 is stored in a storage device provided in the photographing device 10, when the stored image is displayed on the display unit 20, when a high quality image is used for image analysis processing, etc.

[0303] The subsequent processing is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0304] As described above, the acquisition unit 401 according to the present embodiment acquires, as an input image, a medium quality image that is an image generated using a plurality of images of a specific part of a subject. In this case, since the input image is a clearer image, the image quality improvement engine can generate a high quality image with higher accuracy. Note that the number of low quality images used to generate a medium quality image may be less than the number of images used to generate a conventional superimposed image.

[0305] The medium quality image is not limited to an image obtained by superimposing a plurality of low quality images, but may be, for example, an image obtained by applying a MAP estimation process to a plurality of low quality images, or an image obtained by synthesizing a plurality of low quality images. When synthesizing a plurality of low quality images, images having different gradations may be synthesized.

[0306] Also, the output unit 405 may output the generated high quality image to the imaging device 10 or another device connected to the image processing device 400, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high performance imaging device, high cost processing, and noise reduction processing may be used.

[0307] <Fifteenth embodiment> Next, an image processing device according to a fifteenth embodiment will be described with reference to Figs. 4 and 5. In this embodiment, an image quality improving unit performs image quality improvement according to the first embodiment and the like as well as image size increase (size increase) of an input image.

[0308] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0309] The acquisition unit 401 according to the present embodiment acquires an image of low image size (low-size image) as an input image. The low-size image is an image that has fewer pixels constituting it than an image of high image size (high-size image) output by a high image quality engine described later. Specifically, for example, when the image size of the high-size image is 1024 pixels wide, 1024 pixels high, and 1024 pixels deep, the image size of the low-size image is 512 pixels high, 512 pixels deep, and the like. In this regard, in this specification, "enhancing the image size" refers to a process of increasing the number of pixels per image and enlarging the image size.

[0310] The image quality improvement unit 404 according to this embodiment is provided with an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast of an input image, and to increase the image size of the input image. Therefore, the image quality improvement engine of this embodiment is configured to input a low-size image and output a high-size image.

[0311] In relation to this, for a group of pairs constituting the teacher data of the image quality improvement engine, the input data constituting each pair is a low-size image, and the output data is a high-size image. The high-size images used for the output data can be obtained from a device with higher performance than the imaging device that obtained the low-size images, or can be obtained by changing the settings of the imaging device. In addition, when a group of high-size images already exists, the group of high-size images may be reduced to an image size of an image expected to be obtained from the imaging device 10, thereby obtaining a group of low-size images to be used as input data. In addition, the high-size images are obtained by superimposing low-size images as in the first embodiment and the like.

[0312] Note that the enlargement of the image size of the input image by the image quality improving unit 404 according to this embodiment is different from simple image enlargement because the image size is obtained as teacher data from a device with higher performance than the image capture device 10 or by changing the settings of the image capture device 10. Specifically, the process of enlarging the image size of the input image by the image quality improving unit 404 according to this embodiment can reduce degradation in resolution compared to the case of simply enlarging the image.

[0313] With this configuration, the image quality improving section 404 according to this embodiment can reduce noise and enhance contrast on an input image, and generate a high-image-quality image with a large image size.

[0314] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5. Note that the processing of steps S520, S530, and S550 according to this embodiment is similar to those in the first embodiment, and therefore the description will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0315] In step S510, the acquisition unit 401 acquires a low-size image from the photographing device 10 or another device as input data to be processed. The acquisition unit 401 may acquire a low-size image generated by the photographing device 10 as an input image in response to a request from the photographing device 10. Such a request may be issued, for example, when the photographing device 10 generates an image, before or after the image generated by the photographing device 10 is stored in a storage device provided in the photographing device 10, when the stored image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0316] The processes in steps S520 and S530 are the same as those in the first embodiment, and therefore the description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image in which noise has been reduced and contrast has been enhanced as a high-quality image, and in which the image size has been increased. The subsequent processes are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0317] As described above, the image quality improving unit 404 according to this embodiment generates a high-quality image in which at least one of noise reduction and contrast enhancement is performed as compared with an input image, and the image size is enlarged. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or labor of the photographer or the subject.

[0318] In this embodiment, a high-quality image is generated by performing the high-quality image processing and the high-resolution processing according to the first embodiment, etc., using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs the image quality improvement processing according to the first embodiment, etc., and another image quality improvement engine that performs the high image size processing.

[0319] In this case, the image quality improvement engine performing the image quality improvement process according to the first embodiment etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment etc. Furthermore, as input data of the teacher data of the image quality improvement engine performing the high image size process, the high image quality image generated by the image quality improvement engine according to the first embodiment etc. is used. Furthermore, as output data of the teacher data of the image quality improvement engine, the high image quality image generated by the image quality improvement engine according to the first embodiment etc. for an image acquired by a high performance imaging device is used. Thereby, the image quality improvement engine performing the high image size process can generate a final high image quality image by increasing the image size of the high image quality image that has been subjected to the image quality improvement process according to the first embodiment etc.

[0320] Moreover, the image resizing process by the image quality improvement engine may be performed before the image quality improvement process by the image quality improvement engine according to the first embodiment, etc. In this case, the teacher data for the image quality improvement engine that performs the image resizing process is configured as a pair group of input data that is a low-size image acquired by the imaging device and output data that is a high-size image. Also, the teacher data for the image quality improvement engine that performs the image quality improvement process according to the first embodiment, etc. is configured as a pair group of input data that is a high-size image and output data that is an image in which the high-size image is superimposed.

[0321] Even with this configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the image size has been enlarged.

[0322] In this embodiment, the image quality improvement process according to the first embodiment and the like has been described with respect to a configuration in which the superimposed image is used as output data of the teacher data, but the output data is not limited to this, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0323] Note that the output unit 405 may output the generated high-quality image to the photographing device 10 or another device connected to the image processing device 400, similarly to the first embodiment.

[0324] <Sixteenth embodiment> Next, an image processing device according to a sixteenth embodiment will be described with reference to Figures 4 and 5. In this embodiment, an image quality improving section improves spatial resolution in addition to the image quality improvement according to the first embodiment and the like.

[0325] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0326] The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. Note that the low spatial resolution image is an image with a lower spatial resolution than the high spatial resolution image output by the image quality improvement unit 404.

[0327] The image quality improvement unit 404 includes an image quality improvement engine, as in the first embodiment. However, the image quality improvement engine of this embodiment is configured to reduce noise and enhance contrast of an input image, and to increase the spatial resolution of the input image. Therefore, the image quality improvement engine according to this embodiment is configured to input a low spatial resolution image and output a high spatial resolution image.

[0328] In relation to this, for the pair group constituting the teacher data of the image quality improvement engine, the input data constituting each pair is a low spatial resolution image, and the output data is a high spatial resolution image. Note that the high spatial resolution image can be obtained from a device with higher performance than the imaging device that obtained the low spatial resolution image, or by changing the settings of the imaging device. Moreover, as for the high spatial resolution image, an image obtained by superimposing a low spatial resolution image is used, similarly to the first embodiment and the like.

[0329] With this configuration, the image quality improving section 404 according to this embodiment can reduce noise and enhance contrast on an input image, and generate a high-image-quality image with high spatial resolution.

[0330] Next, a series of image processing according to this embodiment will be described with reference to Fig. 5. Note that the processing of steps S520, S530, and S550 according to this embodiment is similar to those in the first embodiment, and therefore the description will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S520, the processing of step S530 may be omitted and the processing may proceed to step S540.

[0331] In step S510, the acquisition unit 401 acquires a low spatial resolution image as input data to be processed from the image capturing device 10 or another device. The acquisition unit 401 may acquire a low spatial resolution image generated by the image capturing device 10 as an input image in response to a request from the image capturing device 10. Such a request may be issued, for example, when the image capturing device 10 generates an image, before or after the image generated by the image capturing device 10 is stored in a storage device provided in the image capturing device 10, when the stored image is displayed on the display unit 20, when a high-quality image is used for image analysis processing, etc.

[0332] The processes in steps S520 and S530 are the same as those in the first embodiment, and therefore the description thereof will be omitted. In step S540, the image quality improvement unit 404 inputs the input image to an image quality improvement engine, and generates an image with high spatial resolution and noise reduction as a high-quality image, in which contrast is enhanced. The subsequent processes are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0333] As described above, the image quality improving unit 404 according to the present embodiment generates an image with improved spatial resolution and at least one of noise reduction and contrast enhancement compared to an input image as a high-quality image. This allows the image processing device 400 according to the present embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or labor of the photographer or subject.

[0334] In this embodiment, a high-quality image is generated by performing the high-quality image processing and the high-resolution processing according to the first embodiment, etc., using one high-quality image engine, but the configuration for performing these processes is not limited to this. For example, the image quality improvement unit may include an image quality improvement engine that performs the image quality improvement processing according to the first embodiment, etc., and another image quality improvement engine that performs the resolution improvement processing.

[0335] In this case, the image quality improvement engine performing the image quality improvement process according to the first embodiment etc. can use a machine learning model that has been trained in the same way as the image quality improvement engine according to the first embodiment etc. Also, as input data of the teacher data of the image quality improvement engine performing the resolution improvement process, the high-image quality image generated by the image quality improvement engine according to the first embodiment etc. Also, as output data of the teacher data of the image quality improvement engine, the high-image quality image generated by the image quality improvement engine according to the first embodiment etc. for an image acquired by a high-performance imaging device is used. As a result, the image quality improvement engine that performs high spatial resolution processing can generate a final high image quality image by increasing the spatial resolution of the high image quality image that has been subjected to the image quality improvement processing related to the first embodiment, etc.

[0336] Furthermore, the high spatial resolution processing by the image quality improvement engine may be performed before the high spatial resolution processing by the image quality improvement engine according to the first embodiment etc. In this case, the teacher data for the image quality improvement engine that performs the high spatial resolution processing is configured from a pair group of input data that is a low spatial resolution image acquired by an imaging device and output data that is a high spatial resolution image. In addition, the training data for the image quality improvement engine that performs the image quality improvement processing related to the first embodiment, etc. is composed of a group of pairs of input data of a high spatial resolution image and output data of an image in which the high spatial resolution image is overlaid.

[0337] Even with such a configuration, the image processing device 400 can generate a high-quality image in which at least one of noise reduction and contrast enhancement has been performed compared to the input image, and the spatial resolution has been improved.

[0338] In this embodiment, the image quality improvement process according to the first embodiment and the like has been described with respect to a configuration in which the superimposed image is used as output data of the teacher data, but the output data is not limited to this, as in the first embodiment. That is, a high-quality image obtained by performing at least one of a group of processes or imaging methods, such as superimposition processing, MAP estimation processing, smoothing filter processing, tone conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing, may be used.

[0339] Furthermore, the image quality improvement unit 404 may use an image quality improvement engine to perform the image quality improvement process according to the fifteenth embodiment in addition to the high spatial resolution process. In this case, at least one of noise reduction and contrast enhancement is performed compared to the input image, and an image with a larger image size and higher spatial resolution compared to the input image can be generated as a high-quality image. This allows the image processing device 400 according to this embodiment to output a high-quality image suitable for image diagnosis without increasing the invasiveness or effort of the photographer or subject.

[0340] Note that the output unit 405 may output the generated high-quality image to the photographing device 10 or another device connected to the image processing device 400, similarly to the first embodiment.

[0341] <Seventeenth embodiment> Next, an image processing apparatus according to a seventeenth embodiment will be described with reference to Figures 22 and 23. In this embodiment, an analysis unit performs image analysis on a high-quality image generated by an image quality improvement unit.

[0342] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0343] 22 shows a schematic configuration of an image processing device 2200 according to this embodiment. In addition to the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 2200 according to this embodiment is provided with an analysis unit 2208. The image processing device 2200 may be composed of a plurality of devices provided with some of these components. Here, the acquisition unit 401, the shooting condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as the configuration of the image processing device according to the first embodiment, so the configuration shown in FIG. 4 is indicated by the same reference numerals and description will be omitted.

[0344] The analysis unit 2208 applies a predetermined image analysis process to the high-quality image generated by the image quality improvement unit 404. In the field of ophthalmology, the image analysis process includes any existing image analysis process such as retinal layer segmentation, layer thickness measurement, optic nerve 3D shape analysis, lamina cribrosa analysis, blood vessel density measurement of OCTA images, and corneal shape analysis, for example, for images acquired by OCT. In addition, the image analysis process is not limited to analysis processes in the field of ophthalmology, and also includes any existing analysis process in the field of radiology, such as diffusion tensor analysis and VBL (Voxel-based Morphometry) analysis.

[0345] The output unit 405 can display the high-quality image generated by the image quality improving unit 404 on the display unit 20, and can also display the analysis result of the image analysis process by the analysis unit 2208. The output unit 405 can display only the image analysis result by the analysis unit 2208 on the display unit 20, or can output the image analysis result to the imaging device 10, an image management system, or other devices. The display form of the analysis result may be any depending on the image analysis process performed by the analysis unit 2208, and may be displayed as an image, a numerical value, or a character, for example. The display form of the analysis result may be an analysis result obtained by analyzing the high-quality image and superimposed on the high-quality image with any transparency. That is, the display form of the analysis result may be an image (for example, a two-dimensional map) obtained by blending the analysis result obtained by analyzing the high-quality image and the high-quality image with any transparency.

[0346] Hereinafter, a series of image processing according to this embodiment will be described with reference to Fig. 23, taking an En-Face image of OCTA as an example. Fig. 23 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S2310 to S2340 according to this embodiment is similar to the processing of steps S510 to S540 in the first embodiment, and therefore the description will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S2320, the processing of step S2330 may be omitted, and the processing may proceed to step S2340.

[0347] In step S2340, the image quality improvement unit 404 improves the image quality of the OCTA En-Face image, and the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high-image-quality image generated in step S2340. As an image analysis of the high-image-quality OCTA En-Face image, any binarization process can be applied to detect a portion corresponding to a blood vessel (blood vessel region) from the image. The area density can be analyzed by determining the ratio of the detected blood vessel portion to the image. In addition, by thinning the binarized blood vessel portion to an image with a line width of 1 pixel, the ratio of the blood vessel that does not depend on the thickness (also called skeleton density) can be determined. Using these images, the area and shape (circularity, etc.) of the avascular zone (FAZ) may be analyzed. As a method of analysis, the above-mentioned numerical values ​​may be calculated from the entire image, or a numerical value may be calculated for a specified region of interest (ROI) based on the instructions of the examiner (user) using a user interface (not shown). The setting of the ROI may not only be specified by the examiner, but may be a predetermined region that is automatically specified. Here, the various parameters described above are examples of analysis results related to blood vessels, and any parameters related to blood vessels may be used. The analysis unit 2208 may perform multiple image analysis processes. That is, an example of analysis of an En-Face image of OCTA is shown here, but not only this, but also retinal layer segmentation, layer thickness measurement, optic disc three-dimensional shape analysis, lamina cribrosa analysis, etc. may be performed on an image acquired by OCT at the same time. In relation to this, the analysis unit 2208 may perform a part or all of multiple image analysis processes in response to an instruction from the examiner via an arbitrary input device.

[0348] In step S2360, the output unit 405 displays the high-quality image generated by the image quality improving unit 404 and the analysis result by the analysis unit 2208 on the display unit 20. The output unit 405 may output the high-quality image and the analysis result to separate display units or devices. The output unit 405 may display only the analysis result on the display unit 20. Furthermore, when the analysis unit 2208 outputs a plurality of analysis results, the output unit 405 may output a part or all of the plurality of analysis results to the display unit 20 or another device. For example, the analysis result regarding blood vessels in the OCTA En-Face image may be displayed on the display unit 20 as a two-dimensional map. Furthermore, a value indicating the analysis result regarding blood vessels in the OCTA En-Face image may be superimposed on the OCTA En-Face image and displayed on the display unit 20.

[0349] As described above, the image processing device 2200 according to this embodiment further includes the analysis unit 2208 that performs image analysis on the high-quality image, and the output unit 405 causes the display unit 20 to display the analysis result by the analysis unit 2208. In this manner, the image processing device 2200 according to this embodiment uses a high-quality image for image analysis, and therefore the accuracy of the analysis can be improved.

[0350] Also, the output unit 405 may output the generated high quality image to the imaging device 10 or another device connected to the image processing device 2200, as in the first embodiment. Also, the output data of the teacher data of the image quality improvement engine is not limited to a high quality image that has been subjected to overlay processing, as in the first embodiment. That is, a high quality image obtained by performing at least one of a group of processes or imaging methods such as overlay processing, MAP estimation processing, smoothing filter processing, gradation conversion processing, imaging using a high performance imaging device, high cost processing, and noise reduction processing may be used.

[0351] <Eighteenth embodiment> Next, an image processing device according to an eighteenth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which a high image quality unit generates a high image quality image by adding noise to an image during learning and learning the noise component.

[0352] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0353] The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or other devices. An example of the configuration of a CNN in the image quality improvement unit according to this embodiment will be described with reference to FIG. 24. FIG. 24 shows an example of a machine learning model configuration in the image quality improvement unit 404. The configuration shown in FIG. 24 is configured with a plurality of layers that process and output an input value group. As shown in FIG. 24, the types of layers included in the configuration include a convolution layer, a downsampling layer, an upsampling layer, and a merging layer. The convolution layer is a layer that performs a convolution process on an input value group according to parameters such as a set filter kernel size, the number of filters, a stride value, and a dilation value. The number of dimensions of the kernel size of the filter may also be changed according to the number of dimensions of the input image. The downsampling layer is a process that reduces the number of output value groups to the number of input value groups by thinning out or merging the input value groups. Specifically, for example, there is Max Pooling processing. The upsampling layer is a process that makes the number of output value groups greater than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. Specifically, for example, there is linear interpolation processing. The synthesis layer is a layer that inputs value groups such as the output value group of a certain layer and the pixel value group constituting an image from multiple sources and performs processing to combine them by connecting or adding them. In such a configuration, the value group output from the pixel value group constituting the input image Im2410 through the convolution processing block and the pixel value group constituting the input image Im2410 are synthesized in the synthesis layer. After that, the synthesized pixel value group is shaped into a high-quality image Im2420 in the last convolution layer. Although not shown, as an example of a modification of the CNN configuration, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer.

[0354] The image quality improvement engine of this embodiment inputs a low quality image obtained by adding a first noise component to an image obtained from the image capture device 10 or another device, and performs training using an image obtained by adding a second noise component to an image obtained from the image capture device 10 or another device as output data as a high quality image. That is, the teacher image used during learning in this embodiment is a common image for the low quality image and the high quality image, and the noise components in each image are different. Since the same image is used, alignment is not required when pairing images.

[0355] The noise components to be added include Gaussian noise, modeled noise specific to the target image, etc. However, the first and second noises are different from each other. Different noise means that the spatial location (pixel position) where the noise is added is different, or the noise value is different. For example, in the case of OCT, noise specific to the target image can be estimated based on data captured without a model eye or a test eye, and used as a noise model. In the case of OCTA, noise that appears in the avascular zone (FAZ) or noise that appears in an image captured of a model eye that reproduces blood flow in a schematic manner can be used as a noise model.

[0356] In the case of Gaussian noise, the standard deviation or variance value is defined as the magnitude of the noise, and noise is randomly added to the image based on these values. As a result of adding random noise, the average value as a whole may not change. That is, the average value of the noise added to each pixel of one image is set to 0. Here, it is not necessary to make the average value 0, and it is sufficient that noise of different patterns can be added to the input data and the output data. In addition, it is not necessary to add noise to both the input data and the output data, and noise may be added to either one of them. Here, when noise is not added, for example, a false image of blood vessels may occur in the image after the image quality is improved, but it is also possible to think that this occurs when the difference between the images before and after the image quality is improved is relatively large. For this reason, the difference between the images before and after the image quality is improved may be reduced. At this time, during learning, two images obtained by adding noise of different patterns to a low image quality image and a high image quality image may be paired images, or two images obtained by adding noise of different patterns to a high image quality image may be paired images.

[0357] The output unit 405 causes the display unit 20 to display the high quality image generated by the image quality improving unit 404 . The output unit 405 may cause the display unit 20 to display the input image together with the high-quality image.

[0358] The subsequent processing is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0359] In this embodiment, a high-quality image is generated using an image obtained by adding a first noise component and a second noise component different from the first noise component to a low-quality image obtained from the photographing device 10 or another device, but the configuration for performing these processes is not limited to this. For example, the image to which noise is added may be a high-quality image that has been subjected to the overlay processing shown in the first embodiment, to which the first and second noise components are added. In other words, a configuration may be used in which an image obtained by adding a first noise component to an overlay processing image is learned as a low-quality image, and an image obtained by adding a second noise component to an overlay processing image is learned as a high-quality image.

[0360] Furthermore, in this embodiment, an example of learning using the first and second noise components has been described, but the present invention is not limited to this. For example, learning may be performed by adding the first noise component only to the image to be made into a low-quality image, and without adding the noise component to the image to be made into a high-quality image. The image in this case may be an image obtained from the photographing device 10 or another device, or an image obtained by superimposing the image may be the target.

[0361] Furthermore, the size of the noise component may be dynamically changed for each type of input image or rectangular area image to be learned. Specifically, adding a large noise value increases the effect of noise removal, while adding a small noise value decreases the effect of noise removal. Therefore, learning may be performed by adjusting the noise to be added according to the conditions and type of the entire image or rectangular area image, for example, by reducing the value of the noise component to be added when the image is dark and increasing the value of the noise component to be added when the image is bright.

[0362] In this embodiment, the image capturing conditions are not specified, but learning is performed using images with various capturing ranges and different numbers of scans, front images of different capturing regions and different depths, etc.

[0363] In the above, we have described images obtained from the photographing device 10 or other devices, noise images obtained by adding noise to those images, superimposition processed images, and images obtained by adding noise to the superimposition processed images. However, the combinations are not limited to those described above, and low-quality images and high-quality images may be combined in any way.

[0364] <Nineteenth embodiment> Next, an image processing device according to a 19th embodiment will be described with reference to Figs. 25 and 26. In this embodiment, an image quality improvement unit includes a plurality of image quality improvement engines, and generates a plurality of high-image-quality images for an input image. An example in which a synthesis unit 2505 synthesizes the plurality of high-image-quality images output from the plurality of image quality improvement engines will be described.

[0365] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0366] The acquisition unit 401 according to this embodiment acquires images as input data to be processed from the image capture device 10 or other devices.

[0367] The image quality improving unit 404 according to this embodiment is provided with a plurality of image quality improving engines, as in the second embodiment. Here, each of the plurality of image quality improving engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, noise components, and image resolution. The image quality improving unit 404 generates a high-quality image using a plurality of image quality improving engines corresponding to at least one of the imaging region, imaging angle of view, front images at different depths, noise components, and image resolution of the input image.

[0368] 26 is a flow diagram of a series of image processing according to this embodiment. Note that the processing of steps S2610 and S2620 according to this embodiment is similar to the processing of steps S510 and S520 in the first embodiment, and therefore description thereof will be omitted. Note that, when the image quality of an input image is to be improved unconditionally with respect to the shooting conditions, after the processing of step S2620, the processing of step S2630 may be omitted and the processing may proceed to step S2640.

[0369] In step S2620, as in the first embodiment, when the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S2630. In step S2630, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image, as in the second embodiment.

[0370] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engines can handle the input image, the process proceeds to step S2660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any of the image quality improvement engines can handle the input image, the process proceeds to step S2640. Note that, depending on the settings and implementation form of the image processing device 400, step S2640 may be performed, as in the first embodiment, even if it is determined that some shooting conditions cannot be handled by the image quality improvement engines.

[0371] In step S2640, the image quality improving unit 404 inputs the input image acquired in step S2610 to each of the image quality improving engines, and generates a group of high image quality images.

[0372] In step S2650, the synthesis unit 2405 synthesizes some high-quality images from the group of high-quality images generated in step S2640. Specifically, for example, the results of two high-quality images are synthesized: a first image quality improvement engine that has learned using paired images of a low-quality image acquired from the image capture device 10 as shown in the first embodiment and a high-quality image obtained by performing superposition processing such as averaging on an image group acquired by capturing low-quality images multiple times, and a second image quality improvement engine that has learned using paired images in which noise has been added to an image as shown in the 18th embodiment. As a synthesis method, synthesis can be performed using averaging, weighted averaging, or the like.

[0373] In step S2660, the output unit 405 displays the image synthesized in step S2650 on the display unit 20 or outputs it to another device. However, if it is determined in step S2630 that the input image cannot be processed, the output unit 405 outputs the input image as an output image. Note that, if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0374] <20th embodiment> Next, an image processing device according to a twentieth embodiment will be described with reference to Fig. 4. In this embodiment, an example will be described in which an image quality improvement unit uses an output result of a first image quality improvement engine and a second image quality improvement engine generates a high image quality image.

[0375] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0376] The acquisition unit 401 according to this embodiment acquires images as input data to be processed from the image capture device 10 or other devices.

[0377] The image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines similar to those in the first embodiment. The image quality improvement unit according to this embodiment includes a first image quality improvement engine that learns, as output data, a low image quality image acquired from the image capture device 10 or another device, and a medium image quality image generated from the plurality of low image quality images. The image quality improvement unit further includes a second image quality improvement engine that learns, as output data, an image output from the first image quality improvement engine, and an image of higher image quality than the medium image quality image. Note that the medium image quality image is the same as in the fourteenth embodiment, and therefore will not be described.

[0378] The output unit 405 causes the display unit 20 to display the high quality image generated by the image quality improving unit 404 . In addition, the output unit 405 may display the input image on the display unit 20 together with the high-quality image, and in this case, the output unit 405 may display on the display unit 20 that the input image is an image generated from multiple low-quality images.

[0379] Next, a series of image processing steps according to this embodiment will be described with reference to Fig. 5. Note that the processing steps from step S510 to step S530 according to this embodiment are similar to those steps in the first embodiment, and therefore description thereof will be omitted.

[0380] In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image and generate a high-image quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to a first image quality improvement engine, which generates a first high-image quality image with improved image quality. Furthermore, the first high-image quality image is input to a second image quality improvement engine, which obtains a second high-image quality image. The image quality improvement engine generates a high-image quality image that is similar to an image that has been superimposed using the input image, based on a machine learning model that has been machine-learned using teacher data. Therefore, the image quality improvement engine can generate a high-image quality image that has been noise-reduced and contrast-enhanced compared to the input image.

[0381] The subsequent processing is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0382] In this embodiment, a high-quality image is generated using a first high-quality engine that has learned a pair of a low-quality image and a medium-quality image obtained from the photographing device 10 or another device, and a second high-quality engine that has learned a pair of a first high-quality image and a high-quality image, but the configuration for performing these processes is not limited to this. For example, the pair of images learned by the first high-quality engine may be an engine that learns noise as described in the 18th embodiment, and the second high-quality engine may learn a pair of a first high-quality image and a high-quality image. As an opposite configuration, the first high-quality engine may learn a pair of a low-quality image and a medium-quality image, and the second high-quality engine may learn an image with noise added to the first high-quality image.

[0383] Furthermore, both the first image quality improvement engine and the second image quality improvement engine may be engines that learn the noise described in the 18th embodiment. In this case, for example, the first image quality improvement engine learns a pair of images in which the first and second noises are added to a high-quality image generated by the superimposition processing image, and the second image quality improvement engine learns a pair of images in which the first and second noises are added to the first high-quality image generated by the first image quality improvement engine. Note that, although two image quality improvement engines have been described in this embodiment, this is not limiting, and a configuration in which a third and a fourth are further connected and processed may be used. By cleaning up the image used for learning, a network that is easy to generate smoother and sharper images is configured.

[0384] <21st embodiment> Next, an image processing apparatus according to a twenty-first embodiment will be described with reference to Figs. 4 and 27. In the first embodiment, the image quality improving unit 404 includes one image quality improving engine. In contrast, in this embodiment, the image quality improvement unit includes a plurality of image quality improvement engines that perform machine learning using different training data, and generates a plurality of high-image-quality images for an input image.

[0385] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as the image processing device 400 according to the second embodiment. Therefore, the image processing device according to this embodiment will be described below, focusing on the differences from the image processing devices according to the first and second embodiments. Since the configuration of the image processing device according to this embodiment is the same as the configuration of the image processing device according to the first and second embodiments, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0386] The image quality improving unit 404 according to this embodiment is equipped with two or more image quality improving engines that have been subjected to machine learning using different teacher data. Here, a method for creating a teacher data group according to this embodiment will be described. First, a pair group of original images as input data and superimposed images as output data, which are taken with images having various shooting ranges and different numbers of scans, is prepared. Taking OCT and OCTA as an example, for example, a first image group pair in which a 3×3 mm range is taken with 300 A scans and 300 B scans, and a second image group pair in which a 10×10 mm range is taken with 500 A scans and 500 B scans, are taken. At this time, the scan density differs by two times between the first image group pair and the second image group pair. Therefore, these image groups are grouped separately. Then, if there is an image group taken with 600 A scans and 600 B scans in a 6×6 mm range, it is grouped in the same group as the first image group. That is, here, image groups with the same or almost the same scan density (with an error of about 10%) are grouped in the same group.

[0387] Next, the pairs are grouped according to the scan density to create a training data set. For example, a group of teacher data is created such that a first teacher data group is composed of a group of pairs obtained by photographing at a first scanning density, and a second teacher data group is composed of a group of pairs obtained by photographing at a second scanning density.

[0388] Then, machine learning is performed in separate image quality improvement engines using each training data. For example, a group of image quality improvement engines is prepared, such as a first image quality improvement engine corresponding to a machine learning model trained with the first training data and a second image quality improvement engine corresponding to a machine learning model trained with the second training data.

[0389] Since each of these image quality improvement engines uses different teacher data for training the corresponding machine learning model, the degree to which the image quality of the input image can be improved varies depending on the shooting conditions of the image input to the image quality improvement engine. Specifically, the first image quality improvement engine provides a high degree of image quality improvement for an input image acquired at a first scanning density, and a low degree of image quality improvement for an image acquired at a second scanning density. Similarly, the second image quality improvement engine provides a high degree of image quality improvement for an input image acquired at a second scanning density, and a low degree of image quality improvement for an image acquired at a first scanning density.

[0390] On the other hand, there are cases where it is not possible to collect a sufficient number of images with various shooting ranges and different scan densities as training data during learning. In such cases, for such image groups, an image quality improvement engine that has learned noise components is prepared, as shown in the 18th embodiment.

[0391] The image quality improvement engine, which has learned about noise components, is less affected by the scan density at the time of shooting, so it is applied when an image with a scan density that has not been learned is input.

[0392] Since each of the teacher data is composed of a pair group grouped by the scan density, the image quality tendency of the image group constituting the pair group is similar. Therefore, the image quality improvement engine can perform image quality improvement more effectively than the image improvement engine according to the first embodiment at a corresponding scan density. Note that the imaging condition for grouping the pairs of teacher data is not limited to the scan density, and may be an imaging part, images at different depths in the front image, or a combination of two or more of these.

[0393] A series of image processing according to this embodiment will be described below with reference to Fig. 27. Fig. 27 is a flow diagram of a series of image processing according to this embodiment. Note that the processes of steps S2710 and S2720 are similar to steps S510 and S520 according to the first embodiment, and therefore description thereof will be omitted.

[0394] When the shooting conditions of the input image are acquired in step S2720, the process proceeds to step S2730. In step S2730, the image quality improvement possibility determination unit 403 uses the shooting conditions acquired in step S2720 to determine whether or not any of the image quality improvement engines included in the image quality improvement unit 404 can handle the input image.

[0395] If the image quality improvement possibility determination unit 403 determines that the shooting conditions are not met, the process proceeds to step S2770. On the other hand, if the image quality improvement possibility determination unit 403 determines that the shooting conditions are met, the process proceeds to step S2740.

[0396] In step S2740, the image quality improvement unit 404 selects an image quality improvement engine that performs image quality improvement processing from the image quality improvement engine group based on the shooting conditions of the input image acquired in step S2720 and the teacher data information of the image quality improvement engine group. Specifically, for example, for the scan density in the shooting conditions group acquired in step S2720, an image quality improvement engine that has teacher data information related to the scan density and provides a high level of image quality improvement is selected. In the above example, when the scan density is the first scan density, the image quality improvement unit 404 selects the first image quality improvement engine.

[0397] On the other hand, in step S2770, the image quality improving section 404 selects an image quality improving engine that has learned the noise components.

[0398] In step S2750, image quality improvement section 404 generates a high-image-quality image by improving the image quality of the input image using the image quality improvement engine selected in steps S2740 and S2770. After that, in step S2760, output section 405 outputs the high-image-quality image in step S2750 and displays it on display section 20. Note that when displaying the high-image-quality image on display section 20, output section 405 may display that the high-image-quality image was generated using the image quality improvement engine selected by image quality improvement section 404.

[0399] As described above, the image quality improvement unit 404 according to this embodiment includes a plurality of image quality improvement engines that have been trained using different learning data. Here, each of the plurality of image quality improvement engines has been trained using different learning data for at least one of the imaging region, imaging angle of view, front images at different depths, and image resolution. Furthermore, for data for which sufficient correct answer data (output data) could not be collected, learning was performed using noise components. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine corresponding to at least one of these.

[0400] With this configuration, the image processing device 400 according to this embodiment can generate high-quality images more effectively.

[0401] <Twenty-second embodiment> Next, an image processing device according to a 22nd embodiment will be described with reference to Figs. 30 to 32. In this embodiment, a wide-angle image generating unit generates a wide-angle image (panoramic image) using a plurality of high-quality images generated by an image quality improving unit.

[0402] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0403] FIG. 31(a) is a flow diagram of a series of image processing according to this embodiment. In step S3110, the acquisition unit 401 acquires a plurality of images (at least two) as input data from the imaging device 10 or another device. The plurality of images are images of different positions of the same subject (such as the subject's eye), and are images of places where the images do not completely overlap the subject but overlap in part. Taking the case of imaging the subject's eye as an example, the position of the fixation light is changed during imaging, and the subject's eye gazes at the fixation light, so that images of different positions of the same subject's eye can be acquired. Note that, when imaging an image, it is desirable to change the position of the fixation light so that at least about 20% of the overlapping areas between adjacent images are in the same place. FIG. 32(a) shows an example of an En-Face image of OCTA captured by changing the position of the fixation light so that adjacent images partially overlap. FIG. 32(a) shows an example of a case where the position of the fixation light is changed and different positions are captured five times. Although five images are shown as an example in FIG. 32, the number is not limited to five and may be two or more.

[0404] The process of step S3120 according to this embodiment is similar to the process of step S520 according to the first embodiment, and therefore the description will be omitted. When the image quality of the input image is to be improved unconditionally with respect to the shooting conditions, the process of step S3130 may be omitted and the process may proceed to step S3140 after the process of step S3120.

[0405] In step S3120, as in the first embodiment, when the shooting condition acquisition unit 402 acquires a group of shooting conditions for the input image, the process proceeds to step S3130. In step S3130, the image quality improvement possibility determination unit 403 uses the acquired group of shooting conditions to determine whether or not the image quality improvement engine included in the image quality improvement unit 404 can handle the input image, as in the first embodiment.

[0406] If the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot handle a plurality of input images, the process proceeds to step S3160. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine can handle a plurality of input images, the process proceeds to step S3140. Note that, depending on the settings and implementation form of the image processing device 400, step S3140 may be performed, as in the first embodiment, even if the image quality improvement engine determines that some shooting conditions cannot be handled.

[0407] In step S3140, the image quality improving unit 404 processes the multiple input images acquired in step S3110 to generate multiple high-image-quality images.

[0408] In step S3150, the wide-angle image generating unit 3005 synthesizes several high-quality images from the group of high-quality images generated in step S3140. Specifically, an OCTA En-Face image will be described as an example. Although the multiple images do not completely overlap, the OCTA En-Face images are taken so that adjacent images overlap each other in some areas. Therefore, the wide-angle image generating unit 3005 detects overlapping areas from the multiple OCTA En-Face images and performs alignment using the overlapping areas. By transforming the OCTA En-Face images based on the alignment parameters and synthesizing the images, it is possible to generate an OCTA En-Face image with a wider range than a single OCTA En-Face image. At this time, since the multiple OCTA En-Face images that are input have been enhanced in image quality in step S3140, the wide-angle OCTA En-Face image output in step S3150 has already been enhanced in image quality. FIG. 32( b ) shows an example of a wide-angle OCTA En-Face image generated by the wide-angle image generating unit 3005 . FIG. 32(b) is an example generated by aligning the five images shown in FIG. 32(a). FIG. 32(c) shows the positional correspondence between FIG. 32(a) and FIG. 32(b). As shown in FIG. 32(c), Im3210 is placed at the center, and Im3220 to Im3250 are placed around it. Note that, for the OCTA En-Face image, a plurality of OCTA En-Face images can be generated by setting different depth ranges from three-dimensional motion contrast data. Therefore, although an example of a surface image with a wide angle of view is shown in FIG. 32, this is not limiting. For example, alignment may be performed using the surface OCTA En-Face image (Im2910) shown in FIG. 29, and the OCTA En-Face images in other depth ranges may be deformed using the parameters obtained there. Alternatively, a color image is used as the input image for alignment, and a composite color image is generated in which the OCTA En-Face of the surface layer is used as the RG component of the RGB components, and the OCTA En-Face image to be aligned is used as the B component. Then, alignment of a composite color OCTA En-Face image in which layers of multiple depth ranges are combined into one image may be performed. By doing so, by extracting only the B component from the aligned color OCTA En-Face image, a wide-angle OCTA En-Face image in which the target OCTA En-Face image has been aligned can be obtained. Note that the target for image quality improvement is not limited to a two-dimensional OCTA En-Face image, but may also be a three-dimensional OCT or three-dimensional motion contrast data itself. In this case, alignment may be performed with three-dimensional data to generate a wide range of three-dimensional data. A high-quality wide-angle image can be generated by cutting out any cross section (any plane in XYZ is possible) or any depth range (range in the Z direction) from a wide range of three-dimensional data.

[0409] In step S3160, the output unit 405 displays the image synthesized from the multiple images in step S3150 on the display unit 20 or outputs it to another device. However, if it is determined in step S3130 that the input image cannot be processed, the output unit 405 outputs the input image as an output image. Note that, if an input image is specified by the examiner or if the input image cannot be processed, the output unit 405 may display on the display unit 20 that the output image is the same as the input image.

[0410] In this embodiment, a single high-quality wide-angle image is generated by generating high-quality images from a plurality of input images and aligning the high-quality images, but the method of generating a single high-quality image from a plurality of input images is not limited to this. For example, in another example of the image quality improvement process of this embodiment shown in Fig. 31(b), a single wide-angle image may be generated first, and the image quality improvement process may be performed on the wide-angle image to generate a single high-quality wide-angle image.

[0411] This process will be described with reference to FIG. 31(b), but a description of the same process as in FIG. 31(a) will be omitted.

[0412] In step S3121, the wide-angle image generating unit 3005 synthesizes the multiple images acquired in step S3110. The wide-angle image generation is the same as that described in step S3150, except that the input image is an image acquired from the image capture device 10 or another device and is an image prior to image quality enhancement.

[0413] In step S3151, the image quality improving unit 404 executes processing on the high-image-quality image generated by the wide-angle-of-view image generating unit 3005 to generate one high-image-quality wide-angle image.

[0414] With this configuration, the image processing device 400 according to this embodiment can generate a high-quality image with a wide angle of view.

[0415] Regarding the above-mentioned first to twenty-second embodiments, the display of a high-quality image on the display unit 20 by the output unit 405 is basically performed automatically in response to the generation of a high-quality image by the image quality improving unit 404 and the output of the analysis result by the analysis unit 2208. However, the display of a high-quality image may be performed in response to an instruction from the examiner. For example, the output unit 405 may display on the display unit 20 an image selected in response to an instruction from the examiner from among the high-quality image generated by the image quality improving unit 404 and the input image. In addition, the output unit 405 may switch the display on the display unit 20 from the captured image (input image) to a high-quality image in response to an instruction from the examiner. That is, the output unit 405 may change the display of a low-quality image to the display of a high-quality image in response to an instruction from the examiner. In addition, the output unit 405 may change the display of a high-quality image to the display of a low-quality image in response to an instruction from the examiner. Furthermore, the image quality improvement unit 404 may start the image quality improvement process by the image quality improvement engine (input of an image to the image quality improvement engine) in response to an instruction from the examiner, and the output unit 405 may display the high-image quality image generated by the image quality improvement unit 404 on the display unit 20. On the other hand, when an input image is photographed by the imaging device 10, the image quality improvement engine may automatically generate a high-image quality image based on the input image, and the output unit 405 may display the high-image quality image on the display unit 20 in response to an instruction from the examiner. Note that these processes can also be performed on the output of the analysis result. That is, the output unit 405 may change the display of the analysis result of the low-image quality image to the display of the analysis result of the high-image quality image in response to an instruction from the examiner. Also, the output unit 405 may change the display of the analysis result of the high-image quality image to the display of the analysis result of the low-image quality image in response to an instruction from the examiner. Of course, the output unit 405 may change the display of the analysis result of the low-image quality image to the display of the low-image quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the low-quality image to a display of the analysis result of the low-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the analysis result of the high-quality image to a display of the high-quality image in response to an instruction from the examiner. Furthermore, the output unit 405 may change the display of the high-quality image to a display of the analysis result of the high-quality image in response to an instruction from the examiner.In addition, the output unit 405 may change the display of the analysis result of the low quality image to a display of another type of analysis result of the low quality image in response to an instruction from the examiner. In addition, the output unit 405 may change the display of the analysis result of the high quality image to a display of another type of analysis result of the high quality image in response to an instruction from the examiner. Here, the display of the analysis result of the high quality image may be a display in which the analysis result of the high quality image is superimposed on the high quality image with an arbitrary transparency. In addition, the display of the analysis result of the low quality image may be a display in which the analysis result of the low quality image is superimposed on the low quality image with an arbitrary transparency. In this case, the change to the display of the analysis result may be, for example, a change to a state in which the analysis result is superimposed on the displayed image with an arbitrary transparency. In addition, the change to the display of the analysis result may be, for example, a change to a display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with an arbitrary transparency. Furthermore, the image processing device may be configured to start processing by the shooting location estimation engine, the image quality evaluation engine, the authenticity evaluation engine, and the evaluation unit in response to an instruction from the examiner. In addition, with respect to the above-mentioned first to twenty-second embodiments, the display mode in which the output unit 405 displays the high-quality image on the display unit 20 may be arbitrary. For example, the output unit 405 may display the input image and the high-quality image side by side, or may switch between them. Furthermore, the output unit 405 may display the input image and the high-quality image in order according to the imaging site, imaging date and time, the facility where imaging was performed, and the like. Similarly, the output unit 405 may display the image analysis results using the high-quality image in order according to any imaging conditions of the high-quality image and the input image corresponding to the high-quality image. Furthermore, the output unit 405 may display the image analysis results using the high-quality image in order for each analysis item.

[0416] <Twenty-third embodiment> Next, an image processing device according to a 23rd embodiment will be described with reference to Fig. 4, Fig. 29 and Fig. 33. In this embodiment, learning is performed using teacher data consisting of a pair group of output data, which is a high-quality image corresponding to input data. At that time, one image quality improvement engine is generated using multiple high-quality output data generated by multiple image quality improvement engines.

[0417] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment. Since the configuration of the image processing device according to this embodiment is similar to the configuration of the image processing device according to the first embodiment, the configuration shown in FIG. 4 is indicated by the same reference numerals, and description thereof will be omitted.

[0418] The acquisition unit 401 according to this embodiment acquires an image as input data to be processed from the image capture device 10 or other devices. The generation of the image quality improvement engine in the image quality improvement unit 404 according to this embodiment will be described with reference to FIG. 29 and FIG. 33. First, the first learning in this embodiment will be described with reference to FIG. 33(a). FIG. 33(a) shows an example of a group of pairs of multiple input data and output data and multiple image quality improvement engines. Im3311 and Im3312 show a group of pairs of input data and output data. For example, this pair is the group of pairs of the surface layer (Im2910) shown in FIG. 29. And, 3313 shows an image quality improvement engine that has been learned using the group of pairs of Im3311 and Im3312. Note that the learning in FIG. 33(a) may be a method of using a high-quality image generated by superimposition processing as described in the first embodiment, or a method of learning noise components as described in the 18th embodiment. Or a combination of these may be used. Im3321 and Im3322 are pairs of input data and output data, for example, the pair group of the deep layer (Im2920) shown in FIG. 29. Then, 3323 indicates an image quality improvement engine that has been trained on the pair group of Im3321 and Im3322. Similarly, Im3331 and Im3332 are pairs of input data and output data, for example, the pair group of the outer layer (Im2930) shown in FIG. 29. Then, 3333 indicates an image quality improvement engine that has been trained on the pair group of Im3331 and Im3332. That is, in FIG. 33(a), learning is performed for each image. Therefore, for example, in the case of the noise component described in the 18th embodiment, learning can be performed with noise parameters suitable for each image. At this time, the image quality improvement engine can include a machine learning engine obtained using training data in which noise according to the state of at least a part of the area of ​​the medical image is added to the at least a part of the area. Here, the noise according to the state may be, for example, noise having a magnitude according to the pixel values ​​of at least a part of the region. Also, the noise according to the state may be, for example, small noise when at least a part of the region has few characteristics (for example, small pixel values, low contrast, etc.).The noise according to the state may be large noise, for example, when there are many features in at least a part of the region (for example, large pixel value, high contrast, etc.). The image quality improvement engine may include a machine learning engine obtained by using learning data including a plurality of front images to which noise of different magnitudes is added for at least two depth ranges among the plurality of depth ranges. In this case, for example, in a depth range corresponding to a front image with few features (for example, small pixel value), the front image to which small noise is added may be used as learning data. Also, for example, in a depth range corresponding to a front image with many features (for example, large pixel value), the front image to which large noise is added may be used as learning data. Note that, in a depth range corresponding to a front image with medium features, the front image to which medium noise is added may be used as learning data. Here, the plurality of depth ranges may be such that two adjacent depth ranges in the depth direction partially overlap each other.

[0419] Next, the inference of an image in this embodiment will be described with reference to FIG. 33(b). In FIG. 33(b), an image is generated using the image quality improvement engines 3313-3333 that have been trained in FIG. 33(a). For example, when a low-quality surface image Im3310 is input to the image quality improvement engine 3313 that has been trained using a plurality of surface images, the image quality improvement engine outputs a high-quality surface image Im3315. When a low-quality deep image Im3320 is input to the image quality improvement engine 3323 that has been trained using a plurality of deep images, the image quality improvement engine outputs a high-quality deep image Im3325. Similarly, when a low-quality outer layer image Im3330 is input to the image quality improvement engine 3333 that has been trained using a plurality of outer layer images, the image quality improvement engine outputs a high-quality outer layer image Im3335.

[0420] Next, the second learning in this embodiment will be described with reference to FIG. 33(c). FIG. 33(c) shows how one image quality improvement engine 3300 is learned using a plurality of image pair groups of different types. Im3310 indicates a low-quality surface image, Im3315 indicates a pair group of high-quality surface images, Im3320 indicates a low-quality deep image, Im3325 indicates a pair group of high-quality deep images, Im3330 indicates a low-quality outer image, and Im3335 indicates a pair group of high-quality outer images. That is, the image quality improvement engine 3300 is generated using teacher data consisting of a pair group of output data, which is a high-quality image generated using the image quality improvement engine learned in the first learning, and low-quality input data. As described above, the image quality improvement engine 3300 can generate high-quality images suitable for image diagnosis, in which noise is reduced and contrast is high, from various types of input images.

[0421] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404. Note that the output unit 405 may cause the display unit 20 to display the input image together with the high-quality image.

[0422] The subsequent processing is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0423] In this embodiment, the En-Face image of OCTA is described using three layers with different depths, but the type of image is not limited to this, and the En-Face image of OCTA with different depth ranges set by changing the reference layer and offset value may be generated to increase the type. The type of image is not limited to the difference in the depth direction, but may be different for each part. For example, it may be different shooting locations such as the anterior eye part and the posterior eye part. Furthermore, the image is not limited to the En-Face image of OCTA, but may be a luminance En-Face image generated from OCT data. Then, in the first learning, learning may be performed separately, and in the second learning, these En-Face images of OCTA and the luminance En-Face images may be learned together. Furthermore, in addition to the En-Face image, it may be a different shooting device such as a tomographic image, an SLO image, a fundus photograph, or a fluorescent fundus photograph.

[0424] Although an example in which the image quality improvement engine is one by the second learning has been described, it is not necessarily required to have one image quality improvement engine. It is sufficient that the image quality improvement engine is configured to learn using a group of pairs of output data of the image quality improvement engine generated in the first learning and low image quality input data. Furthermore, in the second learning, FIG. 33(c) shows an example in which learning is performed simultaneously using a plurality of image pair groups of different types, but this is not limited to this, and transfer learning may also be used. For example, after learning using a group of pairs of surface images of Im3310 and Im3315, the network may be used to learn a group of pairs of deep images of Im3320 and Im3325, thereby finally generating the image quality improvement engine 3300.

[0425] With this configuration, the image quality improving section 404 according to this embodiment can generate high-image-quality images more effectively for various types of images.

[0426] <Twenty-fourth embodiment> Next, an image processing device according to the 24th embodiment will be described with reference to Fig. 34. In this embodiment, an example will be described in which the output unit 405 displays the processing result in the image quality improvement unit 404 on the display unit 20. Note that, in this embodiment, the description will be given with reference to Fig. 34, but the display screen is not limited to this. The image quality improvement process can also be applied to a display screen that displays multiple images obtained at different dates and times side by side, such as in follow-up observation. The image quality improvement process can also be applied to a display screen, such as an imaging confirmation screen, on which the examiner checks whether imaging was successful immediately after imaging.

[0427] Unless otherwise specified, the configuration and processing of the image processing device according to this embodiment are the same as those of the image processing device 400 according to the first embodiment. Therefore, the following description of the image processing device according to this embodiment will focus on the differences from the image processing device according to the first embodiment.

[0428] The output unit 405 can display a plurality of high quality images generated by the image quality improving unit 404 and low quality images that have not been subjected to image quality improvement on the display unit 20. This makes it possible to output low quality images and high quality images according to the examiner's instructions.

[0429] An example of the interface 3400 will be described below with reference to FIG. 34. 3400 indicates the entire screen, 3401 indicates a patient tab, 3402 indicates an imaging tab, 3403 indicates a report tab, and 3404 indicates a setting tab. The diagonal lines in the report tab 3403 indicate the active state of the report screen. In this embodiment, an example of displaying a report screen will be described. Im3405 indicates an SLO image, and Im3406 indicates an OCTA En-Face image shown in Im3407 superimposed on the SLO image Im3405. Here, the SLO image is a front image of the fundus acquired by an SLO (Scanning Laser Ophthalmoscope) optical system (not shown). Im3407 and Im3408 indicate OCTA En-Face images, Im3409 indicates a luminance En-Face image, and Im3411 and Im3412 indicate tomographic images. 3413 and 3414 are displayed by superimposing the boundary line of the upper and lower ranges of the OCTA En-Face images shown in Im3407 and Im3408, respectively, on the tomographic image. Button 3420 is a button for specifying the execution of image quality improvement processing. Of course, as described later, button 3420 may be a button for instructing the display of a high-quality image.

[0430] In this embodiment, the image quality improvement process is performed by designating the button 3420, or the execution is determined based on information saved (stored) in the database. First, an example in which the display of a high image quality image and the display of a low image quality image are switched by designating the button 3420 in response to an instruction from the examiner will be described. Note that the image to be subjected to the image quality improvement process will be described as an En-Face image of OCTA. When the examiner designates the report tab 3403 and transitions to the report screen, the En-Face images Im3407 and Im3408 of OCTA, which are low image quality, are displayed. After that, when the examiner designates the button 3420, the image quality improvement unit 404 performs image quality improvement process on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement process is completed, the output unit 405 displays the high image quality image generated by the image quality improvement unit 404 on the report screen. In addition, Im3406 displays an image that has been subjected to image quality improvement processing because Im3407 is superimposed on the SLO image Im3405. Then, the display of the button 3420 is changed to an active state, and a display is made so that it is clear that the image quality improvement processing has been performed. Here, the execution of processing in the image quality improvement unit 404 does not need to be limited to the timing when the examiner designates the button 3420. Since the types of the OCTA En-Face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, the image quality improvement processing may be performed when transitioning to the report screen. Then, the output unit 405 may display the high image quality image on the report screen at the timing when the button 3420 is pressed. Furthermore, the types of images to be subjected to image quality improvement processing according to an instruction from the examiner or when transitioning to the report screen do not need to be two types. Processing may be performed on images that are likely to be displayed, for example, multiple OCTA En-Face images such as the superficial layer (Im2910), deep layer (Im2920), outer layer (Im2930), and choroidal vascular network (Im2940) shown in Fig. 29. In this case, the images obtained by performing the image quality improvement processing may be temporarily stored in a memory or a database.

[0431] Next, a case where image quality improvement processing is performed based on information saved (stored) in the database will be described. If the state for performing image quality improvement processing is saved in the database, when transitioning to the report screen is made, a high-image-quality image obtained by performing image quality improvement processing is displayed by default. Then, the button 3420 is displayed in an active state by default, so that the examiner can know that a high-image-quality image obtained by performing image quality improvement processing is being displayed. If the examiner wishes to display a low-image-quality image before image quality improvement processing, the examiner can display the low-image-quality image by designating the button 3420 to cancel the active state. If the examiner wishes to return to a high-image-quality image, the examiner designates the button 3420. The execution of the high image quality processing for the database is specified by hierarchy, such as common to all the data stored in the database, and for each piece of photographed data (each examination). For example, when the state of executing the high image quality processing for the entire database is saved, if the examiner saves a state in which the high image quality processing is not executed for each piece of photographed data (each examination), the photographed data is displayed in a state in which the high image quality processing is not executed the next time the photographed data is displayed. In order to save the execution state of the high image quality processing for each piece of photographed data (each examination), a user interface (e.g., a save button) not shown may be used. In addition, when transitioning to other photographed data (other examination) or other patient data (e.g., changing to a display screen other than the report screen in response to an instruction from the examiner), the execution state of the high image quality processing may be saved based on the display state (e.g., the state of the button 3420). In this way, when the execution of the high image quality processing is not specified for each piece of photographed data (each examination), the processing is performed based on the information specified for the entire database, and when it is specified for each piece of photographed data (each examination), the processing can be performed individually based on the information.

[0432] In this embodiment, an example is shown in which Im3407 and Im3408 are displayed as OCTA En-Face images, but the OCTA En-Face image to be displayed can be changed according to the examiner's designation. Therefore, the change of the image when the execution of image quality improvement processing is designated (button 3420 is active) will be described.

[0433] The image is changed using a user interface (e.g., a combo box) not shown. For example, when the examiner changes the type of image from the superficial layer to the choroidal vascular network, the image quality improvement unit 404 performs image quality improvement processing on the choroidal vascular network image, and the output unit 405 displays the high-quality image generated by the image quality improvement unit 404 on the report screen. That is, the output unit 405 may change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range that is at least partially different from the first depth range, in response to an instruction from the examiner. At this time, the output unit 405 may change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range by changing the first depth range to the second depth range in response to an instruction from the examiner. Note that, for an image that is likely to be displayed at the time of transition to the report screen as described above, if a high-quality image has already been generated, the output unit 405 may display the generated high-quality image. The method of changing the type of image is not limited to the above, and it is also possible to generate an En-Face image of OCTA with a different depth range set by changing the reference layer and offset value. In that case, when the reference layer or offset value is changed, the image quality improvement unit 404 executes image quality improvement processing on an arbitrary En-Face image of OCTA, and the output unit 405 displays the high-quality image on the report screen. The reference layer and offset value can be changed using a user interface (e.g., a combo box or a text box) not shown. In addition, the generation range of the En-Face image of OCTA can be changed by dragging either of the boundary lines 3413 and 3414 superimposed and displayed on the tomographic images Im3411 and Im3412 (moving the layer boundary). When the boundary line is changed by dragging, the execution command of the image quality improvement processing is executed continuously. Therefore, the image quality improvement unit 404 may always process the execution command, or may execute it after the layer boundary is changed by dragging. Alternatively, the image quality improvement process may be continuously instructed, but when the next command is received, the previous command may be cancelled and the latest command may be executed. Note that the image quality improvement process may take a relatively long time. Therefore, regardless of the timing at which the command is executed, it may take a relatively long time until a high-quality image is displayed. Therefore, from when a depth range for generating an OCTA En-Face image is set in response to an instruction from the examiner until a high-quality image is displayed, an OCTA En-Face image (low-quality image) corresponding to the set depth range may be displayed. That is, when the depth range is set, an OCTA En-Face image (low-quality image) corresponding to the set depth range may be displayed, and when the high-quality image processing is completed, the display of the OCTA En-Face image (the low-quality image) may be changed to a display of a high-quality image. Also, from when the depth range is set until a high-quality image is displayed, information indicating that the high-quality image processing is being performed may be displayed. Note that these can be applied not only to a case where the execution of the high-quality image processing has already been specified (the button 3420 is in an active state), but also to, for example, a case where the execution of the high-quality image processing is instructed in response to an instruction from the examiner until a high-quality image is displayed.

[0434] In this embodiment, an example has been shown in which different layers are displayed as OCTA En-Face images in Im3407 and Im3408, and low-image-quality and high-image-quality images are switched between for display, but this is not limiting. For example, a low-image-quality OCTA En-Face image may be displayed side by side in Im3407, and a high-image-quality OCTA En-Face image may be displayed side by side in Im3408. When images are switched and displayed, it is easy to compare parts that have changed because the images are switched at the same location, and when displayed side by side, it is easy to compare the entire images because the images can be displayed simultaneously.

[0435] Next, the execution of the high image quality processing during screen transition will be described with reference to Figs. 34(a) and (b). Fig. 34(b) is an example of a screen in which the OCTA En-Face image Im3407 in Fig. 34(a) is enlarged and displayed. In Fig. 34(b), a button 3420 is displayed in the same manner as in Fig. 34(a). The screen transition from Fig. 34(a) to Fig. 34(b) is performed by, for example, double-clicking the OCTA En-Face image Im3407, and the screen transition from Fig. 34(b) to Fig. 34(a) is performed by pressing the close button 3430. Note that the screen transition is not limited to the method shown here, and a user interface (not shown) may be used. If the execution of the high image quality processing is specified during the screen transition (button 3420 is active), the state is maintained during the screen transition. That is, when a high image quality image is displayed on the screen of Fig. 34(a) and the screen of Fig. 34(b) is transitioned to the screen of Fig. 34(b), the high image quality image is also displayed on the screen of Fig. 34(b). Then, the button 3420 is set to an active state. The same applies when transitioning from Fig. 34(b) to Fig. 34(a). In Fig. 34(b), it is also possible to switch the display to a low-quality image by specifying the button 3420. Regarding screen transitions, the transition is not limited to the screens shown here, and as long as the transition is to a screen that displays the same captured data, such as a display screen for follow-up observation or a display screen for panoramic images, the transition...

Claims

1. an acquisition unit that acquires a first image, which is a medical image of a predetermined part of a subject; an image quality improvement unit that uses an image quality improvement engine including a machine learning engine to generate, from the first image, a second image having image quality improved compared to the first image; a display control unit that displays on a display unit a composite image obtained by combining the first image and the second image at a ratio obtained by using information on at least a partial region of the first image; and A medical image processing device comprising:

2. The medical image processing apparatus according to claim 1 , wherein a ratio of combining the first image and the second image is obtained by using pixel values ​​in at least a partial area of ​​the first image as the information.

3. 3. The medical image processing device according to claim 1, wherein the ratio of combining the first image and the second image is obtained by using a difference value of pixel values ​​in at least a portion of corresponding areas of the first image and the second image as the information.

4. The medical image processing apparatus according to claim 1 , wherein a ratio at which the first image and the second image are combined is changeable in response to an instruction from an examiner.

5. 5. The medical image processing device according to claim 1, wherein the ratio of synthesis of the first image and the second image is determined from the information by using a trained model obtained by training with training data in which a medical image is used as input data and information regarding the ratio of synthesis of the medical image and a medical image obtained by improving the image quality of the medical image is used as correct answer data.

6. The medical image processing apparatus according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained by using learning data to which noise has been added to at least a partial area of ​​a medical image.

7. The medical image processing device according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained using learning data to which noise corresponding to the state of at least a portion of a medical image has been added to the at least a portion of the medical image.

8. an acquisition unit that acquires a first image, which is a medical image of a predetermined part of a subject; an image quality improvement unit that uses an image quality improvement engine including a machine learning engine obtained using learning data in which noise corresponding to a state of at least a portion of a medical image is added to the at least a portion of the medical image, to generate a second image from the first image, the second image having higher image quality than the first image; A medical image processing device comprising:

9. The medical image processing device according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained using learning data in which noise of a magnitude corresponding to pixel values ​​of at least a portion of an area of ​​a medical image is added to the at least a portion of the area.

10. The medical image processing apparatus according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained by using learning data including a plurality of medical images, each of which has noise of a different pattern added thereto, as paired images.

11. 11. The medical image processing device according to claim 1, wherein the image quality improvement engine includes a machine learning engine obtained by using learning data including a plurality of medical images obtained by adding noise of different patterns to a medical image obtained by overlay processing as pair images.

12. A designation unit is further provided for designating a part of a depth range of the predetermined portion in the three-dimensional medical image data of the predetermined portion in response to an instruction from an examiner, The acquisition unit acquires a front image corresponding to the specified partial depth range as the first image, The medical image processing device according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained using learning data including a plurality of front images corresponding to a plurality of depth ranges of a predetermined part of a subject.

13. a designation means for designating a part of a depth range of a predetermined part of a subject in three-dimensional medical image data of the predetermined part in response to an instruction from an examiner; an acquisition means for acquiring a first image, which is a front image of the predetermined portion corresponding to the specified partial depth range, by using the three-dimensional data; an image quality improvement unit that uses a machine learning engine obtained by using learning data including a plurality of front images corresponding to a plurality of depth ranges of a predetermined part of a subject to generate, from the first image, a second image having image quality improved compared to the first image; A medical image processing device comprising:

14. The medical image processing device according to claim 12 or 13, wherein the image quality improvement engine includes a machine learning engine obtained using learning data including the plurality of front images to which noise of different magnitudes has been added for at least two of the plurality of depth ranges.

15. 15. The medical image processing device according to claim 12, further comprising a wide-angle image generation unit that generates a wide-angle image using a plurality of second images obtained from a plurality of first images obtained by photographing different positions of the specified part in a direction intersecting a depth direction of the specified part so that a portion of a region of the plurality of adjacent front images corresponding to the specified partial depth range overlaps.

16. The medical image processing device according to any one of claims 1 to 15, wherein the image quality improvement engine includes a machine learning engine obtained using learning data including an image obtained by OCTA imaging using an OCT imaging device with higher performance than the OCT imaging device used for OCTA imaging of the first image, or an image obtained in an OCTA imaging process that requires more labor than the OCTA imaging process of the first image.

17. 17. The medical image processing device according to claim 1, wherein the image quality improvement unit divides the first image into a plurality of two-dimensional images, inputs the divided images to the image quality improvement engine, and generates the second image by integrating the plurality of output images from the image quality improvement engine.

18. The image quality improvement engine includes a machine learning engine obtained by using learning data including a plurality of medical images having corresponding positional relationships as paired images; The medical image processing apparatus according to claim 17 , wherein the image quality improving unit divides the first image into the plurality of two-dimensional images with an image size corresponding to an image size of the paired images and inputs the divided images to the image quality improving engine.

19. The image quality improvement engine performs the following on an area including a medical image and an external periphery of the medical image: The medical image processing apparatus according to claim 17 or 18, further comprising a machine learning engine obtained using learning data including images of a plurality of partial regions that are set so that adjacent partial regions partially overlap each other.

20. The medical image processing apparatus according to claim 1 , wherein the image quality improvement engine includes a machine learning engine obtained by using learning data including medical images obtained by superimposition processing.

21. acquiring a first image which is a medical image of a predetermined part of a subject; generating a second image from the first image using an image enhancement engine including a machine learning engine, the second image having image enhancement compared to the first image; displaying a composite image obtained by combining the first image and the second image at a ratio obtained using information on at least a partial region of the first image on a display unit; A medical image processing method comprising:

22. acquiring a first image which is a medical image of a predetermined part of a subject; generating, from the first image, a second image having a higher image quality than the first image, using an image quality improvement engine including a machine learning engine obtained using learning data in which noise corresponding to a state of at least a portion of a medical image is added to the at least a portion of the medical image; A medical image processing method comprising:

23. Specifying a depth range of a predetermined portion of the subject's three-dimensional medical image data in response to an instruction from an examiner; acquiring a first image, which is a front image of the predetermined portion corresponding to the specified partial depth range, using the three-dimensional data; generating a second image having a higher image quality than the first image from the first image using a machine learning engine obtained by using learning data including a plurality of front images corresponding to a plurality of depth ranges of a predetermined part of the subject; A medical image processing method comprising:

24. A program which, when executed by a processor, causes the processor to perform each step of the medical image processing method according to any one of claims 21 to 23.

Images