Medical image processing device, medical image processing method and program

Abstract

The present invention provides a medical image processing device, a medical image processing method and a computer-readable medium. The medical image processing device includes: an acquisition unit configured to acquire a first image, which is a motion contrast frontal image corresponding to at least a partial depth range of three-dimensional motion contrast data of an eye to be inspected; and an image quality improvement unit configured to generate a second image from the first image using an image quality improvement engine including a machine learning engine, the second image having undergone image quality improvement compared to the first image.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, a medical image processing method and a program. Background Art

[0002] In the medical field, in order to identify a disease of a subject and observe the extent of the disease, images are acquired by various imaging devices, and image diagnosis is performed by medical professionals. In the field of radiology, different types of imaging devices include, for example, X-ray imaging devices, X-ray computed tomography (CT) devices, magnetic resonance imaging (MRI) devices, positron emission tomography (PET) devices, and single photon emission computed tomography (SPECT) devices. In addition, for example, in the field of ophthalmology, different types of imaging devices include fundus cameras, scanning laser ophthalmoscopy (SLO), optical coherence tomography (OCT) devices, and OCT angiography (OCTA) devices.

[0003] In order to accurately perform image diagnosis and complete image diagnosis in a short time, the quality level of the image acquired by the camera is important, that is, it is important that the image has low noise, high resolution and spatial resolution, and appropriate grayscale. In addition, in some cases, it may also be useful to enhance the image of the desired observation site or lesion.

[0004] However, for many imaging devices, in order to obtain images suitable for image diagnosis, such as images with high image quality, a certain price must be paid. For example, although one method is to purchase a high-performance imaging device to obtain images with high image quality, in most cases, a larger investment is required than purchasing a low-performance imaging device.

[0005] In addition, for example, when CT is used, in order to obtain an image with less noise, it is sometimes necessary to increase the amount of radiation to which the subject is exposed. In addition, for example, when MRI is used, in some cases, a contrast agent with a risk of side effects is used to obtain an image that enhances the part desired to be observed. In addition, for example, when OCT is used, when it is necessary to widen the area to be imaged or to obtain high spatial resolution, in some cases, the imaging time becomes longer. In addition, for example, with some imaging devices, in order to obtain an image with high image quality, it is necessary to acquire images multiple times, and the time required for imaging is therefore increased.

[0006] Patent document 1 discloses a technology that converts a previously acquired image into an image with high resolution by means of an artificial intelligence engine in order to respond to rapid developments in medical technology and also to simple imaging in emergency situations. According to this technology, for example, an image acquired at a low cost by simple imaging can be converted into an image with higher resolution.

[0007] [Citation List]

[0008] [Patent Document]

[0009] Patent Document 1: Japanese Patent Application Publication No. 2018-5841 Summary of the invention

[0010] [Technical issues]

[0011] However, even if an image has high resolution, it cannot be said that the image is an image suitable for image diagnosis in some cases. For example, even when the resolution of an image is high, if there is a lot of noise or low contrast in the image, etc., the object to be observed cannot be properly determined in some cases.

[0012] In this regard, an object of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program that are capable of generating images that are more suitable for image diagnosis than conventional techniques.

[0013] [Solution to the problem]

[0014] According to one embodiment of the present invention, a medical image processing device includes: an acquisition unit, which is configured to obtain a first image, which is a motion contrast frontal image corresponding to at least a partial depth range of three-dimensional motion contrast data of an eye to be inspected; and an image quality improvement unit, which is configured to use an image quality improvement engine including a machine learning engine to generate a second image whose image quality has been improved compared to the first image from the first image.

[0015] In addition, a medical image processing method according to another embodiment of the present invention includes: obtaining a first image, which is a motion contrast frontal image corresponding to at least a partial depth range of three-dimensional motion contrast data of the eye to be inspected; and using an image quality improvement engine including a machine learning engine to generate a second image that has undergone image improvement compared to the first image from the first image.

[0016] [Beneficial effects of the present invention]

[0017] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [ Figure 1 ] Figure 1 is a diagram showing an example of the configuration of a neural network related to image quality improvement processing.

[0019] [ Figure 2 ] Figure 2is a diagram showing an example of the configuration of a neural network related to imaging position estimation processing.

[0020] [ Figure 3 ] Figure 3 is a diagram showing an example of the configuration of a neural network related to authenticity evaluation processing of an image.

[0021] [ Figure 4 ] Figure 4 is a diagram showing an example of a schematic configuration of an image processing apparatus according to the first embodiment.

[0022] [ Figure 5 ] Figure 5 is a flowchart showing an example of the flow of image processing according to the first embodiment.

[0023] [ Figure 6 ] Figure 6 is a flowchart showing a different example of the flow of the image processing according to the first embodiment.

[0024] [ Figure 7 ] Figure 7 is a flowchart showing an example of the flow of image processing according to the second embodiment.

[0025] [ Figure 8 ] Figure 8 is a diagram for describing image processing according to the fourth embodiment.

[0026] [ Fig. 9 ] Fig. 9 is a flowchart showing an example of the flow of image quality improvement processing according to the fourth embodiment.

[0027] [ Fig.10 ] Fig.10 is a diagram for describing image processing according to the fifth embodiment.

[0028] [ Fig.11 ] Fig.11 is a flowchart showing an example of the flow of image quality improvement processing according to the fifth embodiment.

[0029] [ Fig.12 ] Fig.12 is a diagram for describing image processing according to the sixth embodiment.

[0030] [ Fig.13 ] Fig.13 is a flowchart showing an example of the flow of image quality improvement processing according to the sixth embodiment.

[0031] [ Fig.14A ] Fig.14A is a diagram for describing image processing according to the sixth embodiment.

[0032] [ Fig. 14B ] Fig. 14B is a diagram for describing image processing according to the sixth embodiment.

[0033] [ Fig. 14C ] Fig. 14C is a diagram for describing image processing according to the sixth embodiment.

[0034] [ Fig.14D ] Fig.14D is a diagram for describing image processing according to the sixth embodiment.

[0035] [ Fig.15 ] Fig.15 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a seventh embodiment.

[0036] [ Fig.16 ] Fig.16 is a flowchart showing an example of the flow of image processing according to the seventh embodiment.

[0037] [ Fig.17 ] Fig.17 is a diagram showing an example of a user interface according to the seventh embodiment.

[0038] [ Fig.18 ] Fig.18 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a ninth embodiment.

[0039] [ Fig.19 ] Fig.19 is a flowchart showing an example of the flow of image processing according to the ninth embodiment.

[0040] [ Fig. 20 ] Fig. 20 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a twelfth embodiment.

[0041] [ Fig.21A ] Fig.21A is a flowchart showing an example of the flow of image quality improvement processing according to the thirteenth embodiment.

[0042] [ Fig.21B ] Fig.21B is a flowchart showing a different example of the flow of the image quality improvement process according to the thirteenth embodiment.

[0043] [ Fig. 22 ] Fig. 22 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a seventeenth embodiment.

[0044] [ Fig.23] Fig.23 is a flowchart showing an example of the flow of image processing according to the seventeenth embodiment.

[0045] [ Fig.24 ] Fig.24 is a diagram showing an example of the configuration of a neural network related to image quality improvement processing.

[0046] [ Fig.25 ] Fig.25 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a nineteenth embodiment.

[0047] [ Fig.26 ] Fig.26 is a flowchart showing an example of the flow of image processing according to the nineteenth embodiment.

[0048] [ Fig. 27 ] Fig. 27 is a flowchart showing an example of the flow of image processing according to the twenty-first embodiment.

[0049] [ Fig.28A ] Fig.28A is a diagram showing an example of a training image related to the image quality improvement process.

[0050] [ Fig.28B ] Fig.28B is a diagram showing an example of a training image related to the image quality improvement process.

[0051] [ Fig.29A ] Fig.29A is a diagram showing an example of an input image related to the image quality improvement process.

[0052] [ Fig.29B ] Fig.29B is a diagram showing an example of an input image related to the image quality improvement process.

[0053] [ Fig.30 ] Fig.30 is a diagram showing an example of a schematic configuration of an image processing apparatus according to a twenty-second embodiment.

[0054] [ Fig.31A ] Fig.31A is a flowchart showing an example of the flow of image processing according to the twenty-second embodiment.

[0055] [ Fig.31B ] Fig.31B is a flowchart showing an example of the flow of image processing according to the twenty-second embodiment.

[0056] [ Fig.32A ] Fig.32ADetailed Description of the Invention ...

[0057] [ Fig.32B ] Fig.32B is a diagram for describing a wide-angle image according to the twenty-second embodiment.

[0058] [ Fig.32C ] Fig.32C is a diagram for describing a wide-angle image according to the twenty-second embodiment.

[0059] [ Fig.33A ] Fig.33A is a diagram for describing the image quality improvement processing according to the twenty-third embodiment.

[0060] [ Fig.33B ] Fig.33B is a diagram for describing a wide-angle image according to the twenty-second embodiment.

[0061] [ Fig.33C ] Fig.33C is a diagram for describing a wide-angle image according to the twenty-second embodiment.

[0062] [ Fig.34A ] Fig.34A is a diagram showing an example of a user interface according to the twenty-fourth embodiment.

[0063] [ Fig.34B ] Fig.34B is a diagram showing an example of a user interface according to the twenty-fourth embodiment.

[0064] [ Fig.35 ] Fig.35 is a diagram showing an example of a schematic configuration of an image processing apparatus according to the twenty-fifth embodiment.

[0065] [ Fig.36A ] Fig.36A is a diagram showing an example of the configuration of a neural network used as a machine learning engine according to Modification 6.

[0066] [ Fig.36B ] Fig.36B is a diagram showing an example of the configuration of a neural network used as a machine learning engine according to Modification 6.

[0067] [ Fig.37A ] Fig.37A is a diagram showing an example of the configuration of a neural network used as a machine learning engine according to Modification 6.

[0068] [ Fig.37B ] Fig.37B is a diagram showing an example of the configuration of a neural network used as a machine learning engine according to Modification 7.

[0069] [ Fig.38 ] Fig.38 is a diagram showing an example of a user interface according to the twenty-fourth embodiment. DETAILED DESCRIPTION

[0070] Exemplary embodiments of the present invention will now be described in detail according to the accompanying drawings. However, the size, material, shape and relative position of the components described in the following embodiments are not definite and may be changed according to the configuration of the device to which the present invention is applied or according to various conditions. In addition, in different drawings, the same or functionally similar elements are represented by the same reference numerals.

[0071] <Term Explanation>

[0072] First, terms used in this specification will be described.

[0073] In the network described in this specification, each device can be connected using a wired or wireless connection. Here, the connection type for connecting each device in the network includes, for example, a dedicated line, a local area network (hereinafter referred to as "LAN") connection, a wireless LAN connection, an Internet connection, Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0074] The medical image processing apparatus may be composed of two or more apparatuses capable of communicating with each other, or may be composed of a single apparatus. In addition, the various components of the medical image processing apparatus may be implemented by a software module executed by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). In addition, the various components may be implemented by a circuit having a specific function such as an ASIC, etc. In addition, the various components may be implemented by any other hardware or a combination of any software.

[0075] The medical images to be processed by the medical image processing apparatus or the medical image processing method according to the embodiments described below include images obtained using any modality (imaging apparatus or imaging method). The medical images to be processed may include medical images obtained by any imaging apparatus, etc., as well as images created by the medical image processing apparatus or the medical image processing method according to the embodiments described below.

[0076] In addition, the medical image to be processed is an image of a predetermined part of the subject (examinee), and the image of the predetermined part includes at least a part of the predetermined part of the subject. The medical image may also include another part of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. In addition, the medical image may be an image representing the structure (form) of the predetermined part, or may be an image representing the function of the predetermined part. For example, images representing functions include images representing hemodynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. Please note that the predetermined part of the subject can be determined according to the imaging target, and these predetermined parts include organs such as the human eye (examined eye), brain, lungs, intestines, heart, pancreas, kidneys, and liver, as well as any parts such as the head, chest, legs, and arms.

[0077] In addition, the medical image may be a tomographic image of the object, or may be a frontal image. Examples of frontal images include a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fundus image obtained by fluorescence imaging, and an in-face image (en-face image) generated using data about at least a partial range in the depth direction of the imaging target regarding data obtained by OCT (three-dimensional OCT data). Note that the in-face image may be an OCTA in-face image (motion contrast frontal image) generated using data about at least a partial range in the depth direction of the imaging target regarding three-dimensional OCTA data (three-dimensional motion contrast data). In addition, three-dimensional OCT data or three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0078] In addition, the term "imaging device" refers to a device for performing imaging to obtain an image to be used for diagnosis. Examples of imaging devices include a device that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, and a device that obtains an image of a predetermined portion by detecting radiation emitted from the subject. More specifically, examples of imaging devices according to the embodiments described below include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

[0079] Note that a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device may be included as examples of an OCT device. Furthermore, examples of a Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a swept source OCT (SS-OCT) device. Furthermore, an adaptive optical SLO (AO-SLO) device and an adaptive optical OCT (AO-OCT) device using an adaptive optical system or the like may be included as examples of an SLO device or an OCT device, respectively. Furthermore, a polarization-sensitive SLO (PS-OCT) device and a polarization-sensitive OCT (PS-OCT) device for visualizing information related to a polarization phase difference or depolarization, etc. may be included as examples of an SLO device or an OCT device, respectively.

[0080] The term "image management system" refers to a device and system that receives and stores images captured by a camera device and images that have undergone image processing. The image management system can also send images in response to requests from connected devices, perform image processing on stored images, and request other devices to execute requests for image processing. Examples of image management systems include picture archiving and communication systems (PACS). In particular, the image management system according to the embodiments described below includes a database that is also capable of storing various information together with the received images, such as information related to the subject and the shooting time associated with the image. In addition, the image management system is connected to a network, and in response to requests from other devices, can send and receive images, convert images, and send and receive various information associated with the stored images.

[0081] The term "imaging conditions" refers to various information related to the conditions when an image obtained by an imaging device is imaged. The imaging conditions include, for example, information related to the imaging device, information related to the facility where the imaging is performed, information about an inspection related to the imaging, information related to the person performing the imaging, and information related to the subject. The imaging conditions also include, for example, the imaging date and time, the imaging part name, the imaging area, the imaging angle of view, the imaging system, the image resolution and grayscale, the image size, the image filter applied, information about the image data format, and information about the radiation amount. Note that examples of the imaging area may include a peripheral area deviating from a specific imaging part, and an area including a plurality of imaging parts.

[0082] The imaging condition may be stored in a data structure constituting an image, may be stored as imaging condition data separated from the image, or may be stored in a database or image management system associated with the imaging device. Therefore, the imaging condition may be obtained by a process corresponding to a unit for storing the imaging condition of the imaging device. Specifically, for example, the imaging condition may be obtained by analyzing the data structure of an image output by the imaging device, acquiring imaging condition data corresponding to the image, or accessing an interface for obtaining the imaging condition from a database associated with the imaging device.

[0083] Note that depending on the camera device, there may also be a camera condition that cannot be obtained because the camera condition is not stored. Examples of this include a case where the camera device does not have a function for obtaining and storing specific camera conditions, or a case where such a function has been disabled. In addition, for example, there is also a case where the camera condition is not stored because it is a camera condition that is not related to the camera device or the camera. In addition, for example, there is also a case where the camera condition is hidden, encrypted, or cannot be obtained without the right to obtain the camera condition. However, there is a case where even unstored camera conditions can be obtained. For example, the name of the camera part or the camera area can be identified by performing image analysis.

[0084] The term "machine learning model" refers to a model that is trained (learned) in advance using appropriate training data (learning data) for any machine learning algorithm. The training data consists of one or more paired groups of input data and output data (correct answer data). Please note that the format and combination of the input data and output data of the paired groups that constitute the training data can be suitable for the desired construction, for example, one of the pairs can be an image and the other can be a numeric value, or one of the pairs can be composed of multiple image groups and the other can be a string, or both elements in the pair can be images.

[0085] Specifically, an example of training data may include training data consisting of paired groups (hereinafter referred to as "first training data"), wherein each pair includes an image obtained by OCT and an imaging site label corresponding to the relevant image. Note that the imaging site label is a unique numerical value or character string representing a site. In addition, an example of other training data may include training data consisting of paired groups (hereinafter referred to as "second training data"), wherein each pair includes a low-quality image with a large amount of noise obtained by normal OCT imaging and a high-quality image obtained by performing OCT imaging multiple times and performing image quality improvement processing.

[0086] When input data is input to the machine learning model, output data according to the design of the relevant machine learning model is output. For example, the machine learning model outputs output data with a high probability corresponding to the input data according to the tendency of training the machine learning model using training data. In addition, for example, according to the tendency of training the machine learning model using training data, the machine learning model can output the probability corresponding to the input data as a numerical value for each output data. Specifically, for example, when an image acquired by OCT is input to a machine learning model trained with the first training data, the machine learning model outputs the imaging site label of the imaging site photographed in the relevant image, or outputs the probability of each imaging site label. In addition, for example, when a low-quality image with a large amount of noise obtained by normal OCT imaging is input to a machine learning model trained with the second training data, the machine learning model outputs a high-quality image equivalent to an image obtained by performing multiple imaging by OCT and performing image quality improvement processing. Please note that with respect to the machine learning model, from the perspective of maintaining quality, the machine learning model can be constructed so as not to use the output data output by the machine learning model itself as training data.

[0087] In addition, the machine learning algorithm includes techniques related to deep learning such as convolutional neural networks (CNNs). In the techniques related to deep learning, if the parameter settings regarding the layer groups and node groups constituting the neural network are different, in some cases, the degree to which the trend trained using the training data is reproducible in the output data will be different. For example, in a machine learning model of deep learning using the first training data, if more appropriate parameters are set, the probability of outputting a correct imaging part label may be higher. In addition, for example, in a machine learning model of deep learning using the second training data, if more appropriate parameters are set, in some cases, an image with higher image quality can be output.

[0088] Specifically, the parameters in the case of CNN may include, for example, the kernel size of the filter set for the convolution layer, the number of filters, the value of the stride, and the dilation value, and also include the number of nodes output from the fully connected layer. Note that the parameter group and the number of training epochs may be set to values ​​preferred for the utilization form of the machine learning model based on the training data. For example, based on the training data, a parameter group or the number of epochs may be set that enables the correct imaging part label to be output with a higher probability or the image to be output with a higher image quality.

[0089] A method for determining such a parameter group or number of epochs will now be described as an example. First, 70% of the pairing groups constituting the training data are set for training purposes, and the remaining 30% are randomly set for evaluation purposes. Next, the machine learning model is trained using the pairing groups for training purposes, and at the end of each training epoch, the training evaluation value is calculated using the pairing groups for evaluation purposes. The term "training evaluation value" refers to, for example, the average value of a set of values ​​obtained by evaluating the output when the input data included in each pairing is input into the machine learning model being trained and the output data corresponding to the input data through a loss function. Finally, the parameter group and the number of epochs when the training evaluation value is minimized are determined as the parameter group and the number of epochs of the relevant machine learning model. Note that by dividing the pairing groups constituting the training data into pairing groups for training purposes and pairing groups for evaluation purposes, and determining the number of epochs in this way, it is possible to prevent the machine learning model from overlearning relative to the pairing groups used for training.

[0090] The term "image quality improvement engine (learning model for improving image quality)" refers to a module that outputs a high-quality image obtained by improving the image quality of an input low-quality image. Here, the term "improving image quality" as used in this specification refers to converting an input image into an image with an image quality more suitable for image diagnosis, and the term "high-quality image" refers to an image that has been converted into an image with an image quality more suitable for image diagnosis. In addition, the term "low-quality image" refers to an image obtained by photographing without setting any specific settings to obtain high image quality, for example, a two-dimensional image or a three-dimensional image obtained by X-ray photography, CT, MRI, OCT, PET or SPECT, or a three-dimensional CT motion image obtained by continuous photography. Specifically, examples of low-quality images include images obtained by an X-ray imaging device or by CT with a small amount of radiation, images obtained by MRI photography without using a contrast agent, images obtained by OCT photography as a short-time photography, and OCTA images obtained by performing a small number of photography times.

[0091] Furthermore, the content of image quality suitable for image diagnosis depends on what is expected for diagnosis using various image diagnoses. Therefore, although it is impossible to say unconditionally, for example, image quality suitable for image diagnosis includes image quality in which the amount of noise is low, the contrast is high, the imaging target is displayed in colors and grayscales that make the imaging target easy to observe, the image size is large, and the resolution is high. In addition, image quality suitable for image diagnosis may include image quality in which an object or grayscale that does not actually exist and is presented during the image generation process is removed from the image.

[0092] Furthermore, if image analysis is performed using high-quality images with little noise and high contrast, such as blood vessel analysis processing of OCTA images or region segmentation processing of CT or OCT images, analysis can be performed more accurately in many cases than when low-quality images are used. Therefore, high-quality images output by the image quality improvement engine are sometimes useful not only for image diagnosis but also for image analysis.

[0093] In the image processing technology constituting the image quality improvement technology in the embodiment described below, processing using various machine learning algorithms such as deep learning is performed. Note that in the image processing technology discussed, in addition to processing using a machine learning algorithm, any existing processing such as various image filtering processing, matching processing using a database of high-quality images corresponding to similar images, and knowledge-based image processing may be performed.

[0094] In particular, Figure 1 The structure shown in can be used as an example of the structure of a CNN for improving the image quality of a two-dimensional image. The structure of the CNN includes a group of multiple convolution processing blocks 100. Each convolution processing block 100 includes a convolution layer 101, a batch normalization layer 102, and an activation layer 103 using a rectifier linear unit. The structure of the CNN also includes a merging layer 104 and a final convolution layer 105. The output value group of the convolution processing block 100 and the pixel value group constituting the relevant image are connected, added, and merged by the merging layer 104. The final convolution layer 105 outputs a pixel value group constituting a high-quality image Im120 merged by the merging layer 104. In such a structure, the value group obtained when the pixel value group constituting the input image Im110 is output after being processed by the convolution processing block 100 group, and the pixel value group constituting the input image Im110 is merged by the merging layer 104. Thereafter, the merged pixel value group is formed into a high-quality image Im120 by the final convolution layer 105.

[0095] Note that, for example, by setting the number of convolution processing blocks 100 to 16, and as a parameter of the convolution layer 101 group, setting the kernel size of the filter to a width of three pixels and a height of three pixels, and setting the number of filters to 64, a certain image quality improvement effect is obtained. However, in practice, as mentioned in the description of the aforementioned machine learning model, a better parameter group can be set by using training data corresponding to the utilization form of the machine learning model. Note that in the case where it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter can be extended to three dimensions or four dimensions.

[0096] Please note that when using certain image processing techniques such as image processing using CNN, you need to pay attention to the image size. Specifically, you should keep in mind that in order to overcome problems such as the image quality of the peripheral parts of the high-quality image not being sufficiently improved, in some cases, the input low-quality image and the output high-quality image need to have different image sizes.

[0097] Although not specifically described in the embodiments described later in order to provide a clear description, in the case of an image quality improvement engine (an image input to the image quality improvement engine and an image output from the image quality improvement engine require different image sizes), it is assumed that the image size is adjusted in an appropriate manner. Specifically, for an input image, such as an image used in training data for training a machine learning model or an image to be input to the image quality improvement engine, padding is performed, or the peripheral camera areas of the relevant input image are combined together to adjust the image size. Note that, according to the characteristics of the image quality improvement technology, the padded area is filled with a fixed pixel value, or filled with adjacent pixel values, or filled by mirroring, so that image quality improvement can be effectively performed.

[0098] Furthermore, the image quality improvement technology is sometimes performed using only one image processing technology, and is sometimes performed using a combination of two or more image processing technologies. In addition, there is also a case where processing of a set of multiple high image quality technologies is performed in parallel to generate a set of multiple high-quality images, and then a high-quality image with the highest image quality is finally selected as the high-quality image. Note that the selection of the high-quality image with the highest image quality can be performed automatically using an image quality evaluation index, or can be performed by displaying a set of multiple high-quality images on a user interface equipped with any display unit, etc., so that selection can be performed according to the instruction of the inspector (user).

[0099] Note that since there are cases where an input image that has not undergone image quality improvement is suitable for image diagnosis, the input image may be added to the object for selecting the final image. In addition, parameters may be input into the image quality improvement engine together with the low-quality image. For example, a parameter specifying the degree of image quality improvement or a parameter specifying the size of an image filter to be used in the image processing technique may be input into the image quality improvement engine together with the input image.

[0100] The term "camera position estimation engine" refers to a module that estimates the camera part or camera area of ​​an input image. The camera position estimation engine can output the position of the camera part or camera area depicted in the input image, or for each camera part label or camera area label at a desired level of detail, it can output the probability of being the relevant camera part or camera area.

[0101] In some cases, the imaging device does not store the imaging part or the imaging area as the imaging condition, or the imaging device cannot obtain and store the imaging part or the imaging area. There is also a case where, even if the imaging part or the imaging area is stored, the imaging part or the imaging area is not stored at the required level of detail. For example, if only the "posterior segment of the eyeball" is stored as the imaging part, it may not be known whether the detailed position of the imaging part is the "macular area" or the "optic nerve head", or the "macular area and the optic nerve head", or the "other area". In addition, for another example, if only the "breast" is stored as the imaging part, it may not be known in more detail whether it is the "right breast", the "left breast" or "both". Therefore, by using the imaging position estimation engine, the imaging part or the imaging area of ​​the input image in this case can be estimated.

[0102] In the image and data processing technology constituting the estimation method of the imaging position estimation engine, processing using various machine learning algorithms such as deep learning is performed. Note that in the image and data processing technology discussed, in addition to or instead of the processing using the machine learning algorithm, any existing estimation processing, such as natural language processing, matching processing using a database of similar images and similar data, and knowledge-based processing, can be performed. Note that images with labels attached to imaging parts or imaging areas can be used as training data for training a machine learning model established using a machine learning algorithm. In this case, images of the training data are used as input data, and labels of the imaging parts or imaging areas are used as output data.

[0103] In particular, Figure 2 The structure shown in can be used as an example of the structure of a CNN for estimating the imaging position of a two-dimensional image. The structure of the CNN includes a group of multiple convolution processing blocks 200, each of which is composed of a convolution layer 201, a batch normalization layer 202, and an activation layer 203 using a rectifier linear unit. The structure of the CNN also includes a final convolution layer 204, a fully connected layer 205, and an output layer 206. The fully connected layer 205 fully connects the output value group of the convolution processing block 200. In addition, the output layer 206 outputs the probability of each assumed imaging site label with respect to the input image Im210 as an estimation result (Result) 207 using a softmax function. In this structure, for example, if the input image Im210 is an image obtained by imaging the "macular area", the imaging site label corresponding to the "macular area" outputs the highest probability.

[0104] Note that, for example, by setting the number of convolution processing blocks 200 to 16, and as a parameter of the convolution layer 201 group, setting the kernel size of the filter to a width of three pixels and a height of three pixels, and setting the number of filters to 64, the imaging part can be estimated with a certain accuracy. However, in practice, as mentioned in the description of the aforementioned machine learning model, a better parameter group can be set by using training data corresponding to the utilization form of the machine learning model. Note that in the case where it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter can be extended to three dimensions or four dimensions. Note that the estimation method is sometimes performed using only one image and data processing technique, and sometimes performed using a combination of two or more image and data processing techniques.

[0105] The term "image quality assessment engine" refers to a module that outputs an image quality assessment index for an input image. In the image quality assessment processing technology that calculates the image quality assessment index, processing using various machine learning algorithms such as deep learning is performed. Note that in the image quality assessment processing technology, any existing evaluation processing may be performed, such as processing using an image noise measurement algorithm and matching processing using a database of image quality assessment indicators corresponding to similar images or fundus images. Note that this type of evaluation processing may be performed in addition to or instead of processing using a machine learning algorithm.

[0106] For example, the image quality evaluation index can be obtained by using a machine learning model established by a machine learning algorithm. In this case, the paired input data constituting the training data for training the machine learning model is an image group consisting of a low-quality image group and a high-quality image group pre-photographed according to various imaging conditions. In addition, the paired output data constituting the training data for training the machine learning model is, for example, an image quality evaluation index group, which is set by an examiner performing image diagnosis for each image group of the input data.

[0107] The term "authenticity evaluation engine" as used in the specification of the present invention refers to a module that evaluates the presentation of an input image and evaluates with a certain degree of accuracy whether the image is obtained by photographing by a target camera. In the authenticity evaluation processing technology, processing using various machine learning algorithms such as deep learning is performed. Note that in the authenticity evaluation processing technology, in addition to or instead of processing using a machine learning algorithm, any existing evaluation processing such as knowledge-based processing may be performed.

[0108] For example, the authenticity assessment process can be implemented by a machine learning model established using a machine learning algorithm. First, the training data for the machine learning model will be described. The training data includes a pairing group consisting of a high-quality image group that is pre-photographed according to various camera conditions and a label indicating that the relevant image is obtained by photographing by a target camera device (hereinafter referred to as a "real label"). In addition, the training data includes a pairing group consisting of a high-quality image group that is generated by inputting a low-quality image into an image quality improvement engine (a first-level image quality improvement engine) and a label indicating that the relevant image is not obtained by photographing by a target camera device (hereinafter referred to as a "pseudo-label"). When the high-quality image generated by the first-level image quality improvement engine is input to a machine learning model trained using such training data, the machine learning model outputs a pseudo-label.

[0109] In particular, Figure 3 The structure shown in can be used as an example of the structure of a CNN that performs authenticity assessment processing on a two-dimensional image. The structure of the CNN includes a group of multiple convolution processing blocks 300, each of which is composed of a convolution layer 301, a batch normalization layer 302, and an activation layer 303 using a rectifier linear unit. The structure of the CNN also includes a last convolution layer 304, a fully connected layer 305, and an output layer 306. The fully connected layer 305 fully connects the output value group of the convolution processing block 300. In addition, the output layer 306 uses the S (sigmoid) function to output a value 1 (true) representing a true label or a value 0 (false) representing a pseudo label as a result (Result) 307 of the authenticity assessment processing for the input image Im310.

[0110] Note that, for example, by setting the number of convolution processing blocks 300 to 16, and as a parameter of the convolution layer 301 group, setting the kernel size of the filter to a width of three pixels and a height of three pixels, and setting the number of filters to 64, a correct result of the authenticity evaluation process is obtained with a certain accuracy. However, in practice, as mentioned in the description of the aforementioned machine learning model, a better parameter group can be set by using training data corresponding to the utilization form of the machine learning model. Note that in the case where it is necessary to process a three-dimensional image or a four-dimensional image, the kernel size of the filter can be extended to three dimensions or four dimensions.

[0111] In some cases, when a high-quality image generated by an image quality improvement engine (a second-level image quality improvement engine) that performs a higher level of image quality improvement than the first-level image quality improvement engine is input to the authenticity evaluation engine, the authenticity evaluation engine outputs a true label. In other words, although it is not the case that the authenticity evaluation engine can definitely evaluate whether the input image is obtained by photographing by a camera device, the authenticity evaluation engine can evaluate whether the image looks like an image obtained by photographing by a camera device. By utilizing this feature, by inputting the high-quality image generated by the image quality improvement engine into the authenticity evaluation engine, it is possible to evaluate whether the high-quality image generated by the image quality improvement engine has undergone sufficient image quality improvement.

[0112] In addition, the efficiency and accuracy of both the image quality improvement engine and the authenticity assessment engine can be improved by coordinating and training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity assessment engine. In this case, first, the machine learning model of the image quality improvement engine is trained so that a true label is output when the authenticity assessment engine is caused to evaluate a high-quality image generated by the high-image quality engine. In addition, in parallel with this, the machine learning model of the authenticity assessment engine is trained so that a pseudo label is output when the authenticity assessment engine is caused to evaluate an image generated by the image quality improvement engine. In addition, in parallel with this, the machine learning model of the authenticity assessment engine is trained so that a true label is output when the authenticity assessment engine is caused to evaluate an image obtained by a camera device. In this way, the efficiency and accuracy of the image quality improvement engine and the authenticity assessment engine are improved.

[0113] <First Embodiment>

[0114] In the following, reference is made to Figure 4 and Figure 5 A medical image processing apparatus according to a first embodiment is described. Figure 4 is a diagram showing an example of a schematic configuration of an image processing apparatus according to the present embodiment.

[0115] The image processing device 400 is connected to the camera 10 and the display unit 20 via a circuit or a network. The camera 10 and the display unit 20 may also be directly connected. Note that although it is assumed in the present embodiment that these devices are separate devices from each other, some or all of these devices may be integrally formed with each other. In addition, these devices may be connected to any other device via a circuit or a network, and may be integrally formed with any other device.

[0116] The image processing device 400 is provided with an acquisition unit 401, an imaging condition acquisition unit 402, an image quality improvement possibility determination unit 403, an image quality improvement unit 404, and an output unit 405 (display control unit). Please note that the image processing device 400 may be composed of a plurality of devices, each of which is provided with one or more of these components. The acquisition unit 401 may obtain various data and images from the imaging device 10 or other devices, and may obtain input by the inspector through an input device (not shown). A mouse, a keyboard, a touch panel, and any other input device may be used as the input device. In addition, the display unit 20 may be configured as a touch panel display.

[0117] The imaging condition obtaining unit 402 obtains the imaging condition of the medical image (input image) obtained by the obtaining unit 401. Specifically, according to the data format of the medical image, an imaging condition group stored in the data structure constituting the medical image is obtained. Note that in the case where the imaging condition is not stored in the medical image, an imaging information group including the imaging condition group can be obtained from the imaging device 10 or the image management system through the obtaining unit 401.

[0118] The image quality improvement possibility determination unit 403 uses the imaging condition group obtained by the imaging condition acquisition unit 402 to determine whether the relevant medical image can be processed by the image quality improvement unit 404. The image quality improvement unit 404 performs processing to improve the image quality of the medical image that can be processed, thereby generating a high-quality image suitable for image diagnosis. The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improvement unit 404, the input image, and various information, etc. The output unit 405 can also store the generated high-quality image, etc. in a storage device (storage unit) connected to the image processing device 400.

[0119] Next, the image quality improvement unit 404 will be described in detail. An image quality improvement engine is provided in the image quality improvement unit 404. According to the image quality improvement technique of the image quality improvement engine according to the present embodiment, processing using a machine learning algorithm is performed.

[0120] In this embodiment, training data consisting of a pairing group is used to train a machine learning model related to a machine learning algorithm, the pairing group consisting of paired input data and output data, the input data being a low-quality image with a specific imaging condition assumed to be a processing object, and the output data being a high-quality image corresponding to the input data. Please note that the specific imaging condition specifically includes a predetermined imaging part, an imaging system, an imaging angle of view, an image size, etc.

[0121] In the present embodiment, the input data of the training data is a low-quality image obtained by a device of the same model as the imaging device 10 and using the same settings as the imaging device 10. In addition, the output data of the training data is a high-quality image obtained by using the settings and image processing equipped by the device of the same model as the imaging device 10. Specifically, the output data is, for example, a high-quality image (superimposed image) obtained by superimposing a group of images (source images) obtained by performing imaging multiple times, such as addition averaging. Here, OCTA motion contrast data will be described by taking high-quality images and low-quality images as examples. In this case, the term "motion contrast data" refers to data used with OCTA, etc., which is obtained by repeatedly imaging the same position of the imaging target and detecting the time change of the imaging target during imaging. At this time, the OCTA in-plane image (motion contrast frontal image) can be generated by generating a frontal image using data of a desired range in the depth direction of the imaging target among the calculated motion contrast data (an example of three-dimensional medical image data). Note that hereinafter, the number of images obtained when imaging is repeatedly performed at the same position to obtain OCT data is referred to as "NOR" (number of repetitions).

[0122] In this embodiment, we will use Fig.28A and Fig.28B Two different types of methods are described as examples of generating high-quality images and low-quality images through superposition processing.

[0123] The first method involves generating motion contrast data from OCT data obtained by repeatedly imaging the same position of an imaging target, and using Fig.28A This is described using a high-quality image as an example. Fig.28A , reference numeral Im2810 represents three-dimensional motion contrast data, and reference numeral Im2811 represents two-dimensional motion contrast data constituting the three-dimensional motion contrast data. In addition, reference numerals Im2811-1 to Im2811-3 represent OCT tomographic images (B scans) used to generate the motion contrast data Im2811. Here, the term "NOR" refers to Fig.28A The number of OCT tomographic images of Im2811-1 to Im2811-3 in the image is 3, and in the example shown in the figure, NOR is 3. Images Im2811-1 to Im2811-3 are photographed at predetermined time intervals (Δt). Note that the term "same position" refers to a line in the front direction (XY) of the subject's eye, and Fig.28A, corresponding to the position of Im2811. Note that the front direction is an example of a direction intersecting with the depth direction. Since motion contrast data is data obtained by detecting time changes, it is necessary to perform at least two NORs to generate the data. For example, when NOR is 2, one motion contrast data is generated. When NOR is 3, two data are generated in the case where motion contrast data is generated using only OCT obtained at continuous time intervals (first time and second time, second time and third time). When motion contrast data is generated by also using OCT data obtained at separate time intervals (first time and third time), a total of three data are generated. In other words, as NOR increases to three times, four times, ..., the number of motion contrast data at the same position also increases. By aligning a plurality of motion contrast data obtained by repeatedly photographing the same position, and performing a superposition process such as additive averaging on the aligned motion contrast data, motion contrast data with high image quality can be generated. To this end, NOR is set to at least 3 or more, and it is desirable to set NOR to 5 or more. On the other hand, motion contrast data in a state before superposition processing such as additive averaging is performed is taken as an example of a low-quality image corresponding thereto. In this case, when performing a superposition process such as additive averaging, it is desirable to adopt a low-quality image as a reference image. When performing a superposition process, if alignment is performed in advance by changing the position of a target image relative to a reference image or deforming the shape of the target image relative to a reference image, there is almost no spatial displacement between the reference image and the image after the superposition process. Therefore, a pair including a low-quality image and a high-quality image can be easily made. Note that a target image that has undergone an image deformation process for alignment can be adopted as a low-quality image instead of a reference image. By adopting each image (reference image and target image) of a source image group as input data and adopting a corresponding superposition image as output data, a plurality of paired groups can be generated. For example, in the case of obtaining a superposition image from a group of 15 source images, a pair consisting of a first source image in the source image group and the superposition image can be generated, and a pair consisting of a second source image in the source image group and the superposition image can be generated. Therefore, in the case of obtaining a superposition image from a group of 15 source images, 15 paired groups can be generated by using an image in the source image group and the superposition image in each pair. Note that three-dimensional high-image quality data can be generated by repeatedly imaging the same position in the main scanning (X) direction and scanning while moving the imaging position in the sub scanning (Y) direction.

[0124] The second method involves a process of generating a high-quality image by superimposing motion contrast data obtained by photographing the same area of ​​the photographing target multiple times, and using Fig.28BDescribe. Note that the term "same area" refers to an area such as a 3×3 mm area or a 10×10 mm area in the front direction (XY) of the eye to be inspected, and means obtaining three-dimensional motion contrast data in the depth direction including the tomographic image. When the same area is imaged multiple times and superimposed, it is desirable to make NOR two or three times in order to shorten the image taken once. In addition, in order to generate three-dimensional motion contrast data with high image quality, at least two or more three-dimensional data of the same area are obtained. Fig.28B An example of multiple three-dimensional motion contrast data is shown. Fig.28A Similar to the described example, reference numerals Im2820 to Im2840 represent three-dimensional motion contrast data. Alignment processing in the front direction (XY) and the depth direction (Z) is performed using these two or more three-dimensional motion contrast data, and then, after removing the data that will become artifacts from each data, averaging processing is performed. Therefore, a three-dimensional motion contrast data with high image quality from which artifacts are removed can be generated. A high-quality image is obtained by generating an arbitrary plane from the three-dimensional motion contrast data. On the other hand, it is desirable to adopt an arbitrary plane generated by the reference data when performing an overlay process such as additive averaging as a low-quality image corresponding to the high-quality image. As described above for the first method, since there is almost no spatial displacement between the reference image and the image after additive averaging, a pairing including a low-quality image and a high-quality image can be easily produced. Note that an arbitrary plane generated by the target data that has undergone image deformation processing for alignment instead of the reference data can be adopted as a low-quality image.

[0125] In the first method, since the imaging itself is completed after one round of imaging, the burden on the subject is small. However, as the number of NORs increases, the time required for one round of imaging also increases. In addition, in the case where artifacts (such as cloudiness of the eyes or eyelashes) enter while imaging is being performed, a good image is not always obtained. In the second method, since imaging is performed multiple times, the burden on the subject increases slightly. However, the time required for one round of imaging is short, and even if artifacts are generated in one round of imaging, as long as the artifacts are not photographed in another round of imaging, it is possible to finally obtain a clear image with almost no artifacts. When collecting data, take these characteristics into consideration and select any method according to the situation of the subject.

[0126] Although the present embodiment has been described by taking motion contrast data as an example, the present invention is not limited thereto. Since OCT data is obtained in order to generate motion contrast data, the aforementioned method can be performed in the same manner using OCT data. In addition, although the description of tracking processing has been omitted from the present embodiment, it is desirable to perform imaging while tracking the eye to be inspected so as to image the same position or the same area of ​​the eye to be inspected.

[0127] In this embodiment, since a pair consisting of three-dimensional high-quality image data and three-dimensional low-quality image data can be made, any pair of two-dimensional images can be generated from three-dimensional data. Fig.29A and Fig.29B For example, when the target image is used as an OCTA in-plane image, the OCTA in-plane image is generated within a desired depth range based on the three-dimensional data. The term "desired depth range" refers to Fig.28A and Fig.28B An example of an OCTA in-plane image generated in this case is shown in Fig.29AAs shown in . OCTA in-plane images generated in different depth ranges such as the surface layer (Im2910), the deep layer (Im2920), the outer layer (Im2930), and the choroidal vascular network (Im2940) are used as OCTA in-plane images for learning. Note that the types of OCTA in-plane images are not limited to these examples, and the types can be increased by generating OCTA in-plane images in which different depth ranges are set by changing the layer used as a reference and the offset value. When learning, learning can be performed separately for each OCTA in-plane image of different depths, learning can be performed by combining multiple images in different depth ranges (for example, dividing the image between the surface layer side and the depth layer), or learning can be performed by learning OCTA frontal images of all depth ranges together. In the case of a brightness in-plane image also generated by OCT data, similar to the OCTA in-plane image, learning is performed using multiple in-plane images generated from an arbitrary depth range. For example, let us consider a case where the image quality improvement engine includes a machine learning engine that is obtained using learning data including multiple motion contrast frontal images corresponding to different depth ranges of the eye to be inspected. At this time, the obtaining unit can obtain a motion contrast positive image corresponding to a partial depth range among a long depth range including different depth ranges as a first image. In other words, when image quality improvement is performed, a motion contrast positive image corresponding to a depth range that is different from a plurality of depth ranges corresponding to a plurality of motion contrast positive images included in the learning data can be used as an input image. Naturally, when image quality improvement is performed, a motion contrast positive image within the same depth range as the depth range used for learning can be used as an input image. In addition, the partial depth range can be set in response to the examiner pressing an arbitrary button on a user interface, etc., or can be set automatically. Note that the above is not limited to motion contrast positive images, and can also be applied to brightness in-plane images, for example.

[0128] Note that when the image to be processed is a tomographic image, learning is performed using a B-scan image or an OCT tomographic image of a tomographic image as motion contrast data. Fig.29B This is described in Fig.29B , reference numerals Im2951 to Im2953 represent OCT tomographic images. Fig.29BThe reason why the images in are different is that they show tomographic images at different positions in the sub-scan (Y) direction. The tomographic images can be used together for learning without paying attention to the differences in the positions in the sub-scan direction. However, in the case of images obtained by imaging different positions of the imaging part (for example, the center of the macular area and the center of the optic nerve head), a structure in which learning is performed separately for each part may be adopted, or a structure in which learning is performed together without paying attention to the imaging part may be adopted. Note that since the image feature values ​​between the OCT tomographic image and the tomographic image of the motion contrast data are significantly different, it is better to learn these images separately.

[0129] Because the pixels that are usually visible in the source image group are enhanced, the superimposed image after the superimposition processing is a high-quality image suitable for image diagnosis. In this case, since the pixels that are usually visible are enhanced, the generated high-quality image is a high-contrast image with a clear difference between low-brightness areas and high-brightness areas. In addition, for example, in the superimposed image, the random noise generated in each round of imaging can be reduced, and another source image group can be used to interpolate the area that is not well presented in the source image at a specific time point.

[0130] In addition, in the case where multiple images are needed to construct the input data of the machine learning model, the required number of source image groups can be selected from the source image groups and used as input data. For example, in the case where a superimposed image is obtained from a group of 15 source images, if two images are needed as input data for the machine learning model, 105 (15C2=105) paired groups can be generated.

[0131] Note that, among the pairs constituting the training data, pairs that do not contribute to improving the image quality can be removed from the training data. For example, if the image quality of a high-quality image as output data included in a pair of training data is not suitable for image diagnosis, there is a possibility that the image output by the image quality improvement engine learned using the relevant training data will have an image quality that is not suitable for image diagnosis. Therefore, by removing pairs whose output data has an image quality that is not suitable for image diagnosis from the training data, the possibility that the image quality improvement engine generates an image with an image quality that is not suitable for image diagnosis can be reduced.

[0132] Furthermore, in the case where the average brightness or brightness distribution differs greatly in the paired image groups, there is a possibility that the image quality improvement engine learned using the relevant training data will output an image that is not suitable for image diagnosis and has a brightness distribution that differs greatly from the brightness distribution of the low-quality image. Therefore, pairs of input data and output data with a large difference in average brightness or brightness distribution can be removed from the training data.

[0133] In addition, in the case where the structure or position of the imaging target to be presented in the paired image group is greatly different, there is a possibility that the image quality improvement engine learned using the relevant training data will output an image that is not suitable for image diagnosis and in which the structure or position of the imaging target is presented is greatly different from that of the low-quality image. Therefore, such paired input data and output data can also be removed from the training data: the structure or position of the imaging target to be presented is greatly different between the input data and the output data. In addition, with respect to the image quality improvement engine, from the viewpoint of maintaining quality, the image quality improvement engine can be constructed so as not to use the high-quality image output by the image quality improvement engine itself as training data.

[0134] By using the image quality improvement engine that has undergone machine learning in this manner, when a medical image obtained by one round of imaging is input, the image quality improvement unit 404 can output a high-quality image with improved contrast or reduced noise, etc., through superposition processing. Therefore, the image quality improvement unit 404 can generate a high-quality image suitable for image diagnosis based on a low-quality image as an input image.

[0135] Next, we will refer to Figure 5 The shown flowchart describes a series of image processing operations according to the present embodiment. Figure 5 5 is a flowchart showing a series of image processing operations according to the present embodiment. First, when a series of image processing operations according to the present embodiment is started, the process proceeds to step S510.

[0136] In step S510, the obtaining unit 401 obtains, as an input image, an image captured by the imaging device 10 from the imaging device 10 connected to the obtaining unit 401 through a circuit or a network. Note that the obtaining unit 401 may obtain the input image in response to a request from the imaging device 10. For example, such a request may be issued when the imaging device 10 generates an image, or when an image generated by the imaging device 10 is displayed on the display unit 20 before the image is stored in a storage device included in the imaging device 10, or when a stored image is displayed on the display unit 20 after the image is stored in the storage device, or when image analysis processing is performed using a high-quality image.

[0137] Note that the acquisition unit 401 can obtain data for generating an image from the camera 10, and the image processing device 400 can obtain an image generated based on the relevant data as an input image. In this case, the image processing device 400 can adopt any existing image generation method as an image generation method for generating various images.

[0138] In step S520, the imaging condition obtaining unit 402 obtains the imaging condition group of the input image. Specifically, according to the data format of the input image, the imaging condition obtaining unit 402 obtains the imaging condition group stored in the data structure constituting the input image. Note that, as described above, in the case where the imaging condition is not stored in the input image, the imaging condition obtaining unit 402 may obtain the imaging information group including the imaging condition group from the imaging device 10 or an image management system not shown in the drawings.

[0139] In step S530, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether the image quality of the input image can be improved by the image quality improvement engine included in the image quality improvement unit 404. Specifically, the image quality improvement possibility determination unit 403 determines whether the imaging part, imaging system, imaging angle of view, and image size of the input image match the conditions that can be processed by the image quality improvement engine.

[0140] The image quality improvement possibility determination unit 403 determines all the imaging conditions, and if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine can process the imaging conditions, the process proceeds to step S540. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot process the input image based on the imaging conditions, the process proceeds to step S550.

[0141] Note that, depending on the arrangement and implementation form of the image processing device 400, even if it is determined that the input image cannot be processed based on some conditions among the imaging part, the imaging system, the imaging angle, and the image size, the image quality improvement processing in step S540 may be performed. For example, such processing may be performed under the assumption that the image quality improvement engine can fully process any imaging part of the subject, and such processing may be implemented so that the input data can be processed even if an unknown imaging part is included in the input data. In addition, the image quality improvement possibility determination unit 403 may determine whether at least one condition among the imaging part, the imaging system, the imaging angle, and the image size of the input image matches the condition that the image quality improvement engine can process according to the desired configuration.

[0142] In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image, thereby generating a high-quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the image quality improvement engine so that the image quality improvement engine generates a high-quality image that has undergone image quality improvement. The image quality improvement engine generates a high-quality image obtained by performing a superposition process using the input image based on a machine learning model that performs machine learning using training data. Therefore, compared with the input image, the image quality improvement engine can generate a high-quality image with reduced noise or enhanced contrast.

[0143] Note that, depending on the configuration or implementation of the image processing apparatus 400, the image quality improvement unit 404 may input parameters together with the input image into the image quality improvement engine according to the imaging condition group to adjust the degree of image quality improvement, etc. In addition, the image quality improvement unit 404 may also input parameters consistent with the input of the examiner into the image quality improvement engine together with the input image to adjust the degree of image quality improvement, etc.

[0144] In step S550, if a high-quality image has been generated in step S540, the output unit 405 outputs the high-quality image and causes the high-quality image to be displayed on the display unit 20. On the other hand, in the case where the image quality improvement process cannot be performed in step S530, the output unit 405 outputs the input image and causes the input image to be displayed on the display unit 20. Note that instead of causing the output image to be displayed on the display unit 20, the output unit 405 may cause the output image to be displayed on the camera 10 or other devices, or may store the output image. In addition, depending on the setting or implementation form of the image processing device 400, the output unit 405 may process the output image so that the output image can be utilized by the camera 10 or other devices, or may convert the data format of the output image so that the output image can be sent to an image management system, etc.

[0145] As described above, the image processing device 400 according to the present embodiment includes an acquisition unit 401 and an image quality improvement unit 404. The acquisition unit 401 acquires an input image (first image) as an image of a predetermined part of a subject. The image quality improvement unit 404 generates a high-quality image (second image) that has undergone at least one of noise reduction and contrast enhancement relative to the input image from the input image using an image quality improvement engine including a machine learning engine. The image quality improvement engine includes a machine learning engine for which an image obtained by superposition processing is used as learning data.

[0146] Through this configuration, the image processing device 400 according to the present embodiment can output a high-quality image with reduced noise and / or enhanced contrast based on an input image. Therefore, the image processing device 400 can obtain an image suitable for image diagnosis, such as a clearer image or an image in which the part or lesion to be observed is enhanced, at a lower cost than conventional techniques without increasing the invasiveness to the subject or the labor of the person performing the imaging.

[0147] The image processing apparatus 400 further includes an image quality improvement possibility determination unit 403, which is used to determine whether a high-quality image can be generated using the image quality improvement engine for the input image. The image quality improvement possibility determination unit 403 makes the determination in question based on at least one of the imaging part, the imaging system, the imaging angle of view, and the image size of the input image.

[0148] With this configuration, the image processing apparatus 400 according to the present embodiment can omit input images that cannot be processed by the image quality improving unit 404 from the image quality improving process, and therefore, the processing load of the image processing apparatus 400 and the occurrence of errors can be reduced.

[0149] Note that although in the present embodiment, the output unit 405 (display control unit) is configured so that the generated high-quality image is displayed on the display unit 20, the operation of the output unit 405 is not limited thereto. For example, the output unit 405 may also output the high-quality image to the camera 10 or other devices connected to the image processing device 400. Therefore, the high-quality image may be displayed on the user interface of these devices, may be stored in any storage device, may be used for any image analysis, or may be sent to an image management system.

[0150] In the present embodiment, the image quality improvement possibility determination unit 403 determines whether the input image is an image whose image quality can be improved by the image quality improvement engine, and if the input image is an image whose image quality can be improved, the image quality improvement unit 404 performs processing to improve the image quality. In this regard, in the case where imaging is performed only by the imaging device 10 or the like under imaging conditions that can improve the image quality, the image quality improvement may be performed unconditionally on the image obtained from the imaging device 10. In this case, as Figure 6 As shown, the processing in step S520 and step S530 may be omitted, and step S540 may be performed after step S510.

[0151] In the present embodiment, the output unit 405 is configured to cause the high-quality image to be displayed on the display unit 20. However, the output unit 405 may cause the high-quality image to be displayed on the display unit 20 in response to an instruction from the examiner. For example, in response to the examiner pressing any button on the user interface of the display unit 20, the output unit 405 may display the high-quality image on the display unit 20. In this case, the output unit 405 may switch from displaying the input image to displaying the high-quality image, or may display the high-quality image side by side with the input image.

[0152] In addition, when a high-quality image is displayed on the display unit 20, the output unit 405 may cause a display indicating that the image being displayed is a high-quality image generated by processing using a machine learning algorithm to be displayed together with the high-quality image. In this case, since the user can easily recognize through the relevant display that the displayed high-quality image is not an actual image obtained by camera imaging, misdiagnosis can be reduced and diagnostic efficiency can be improved. Note that the display indicating that a high-quality image is generated by processing using a machine learning algorithm may be in any form as long as the display is a display that makes it possible to distinguish between the input image and the high-quality image generated by the relevant processing.

[0153] Furthermore, regarding a display indicating that a high-quality image is generated by processing using a machine learning algorithm, the output unit 405 may cause a display indicating what kind of training data the machine learning algorithm uses when the machine learning algorithm performs learning to be displayed on the display unit 20. The display in question may include a description of the types of input data and output data of the training data, or any display related to the training data, for example, an imaging site included in the input data and output data.

[0154] Although the superimposed image is used as output data of the training data in the image quality improvement engine according to the present embodiment, the training data is not limited thereto. A high-quality image obtained by performing at least one of the superimposed processing as a unit for obtaining a high-quality image, the processing of the processing group described later, and the imaging method described later can be used as output data of the training data.

[0155] For example, a high-quality image obtained by performing maximum a posteriori processing (MAP estimation processing) on ​​a source image group can be used as output data of training data. In the MAP estimation processing, a likelihood function is obtained based on the probability density of each pixel value in a plurality of low-quality images, and the obtained likelihood function is used to estimate the true signal value (pixel value).

[0156] The high-quality image obtained by the MAP estimation process is a high-contrast image based on a pixel value close to a true signal value. In addition, since the estimated signal value is determined based on the probability density, the randomly generated noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the image quality improvement engine can generate a high-quality image with reduced noise and high contrast suitable for image diagnosis from the input image. Note that regarding the method of generating paired input data and output data for training data, a method similar to the case of using a superimposed image as training data can be performed.

[0157] In addition, as output data of training data, a high-quality image obtained by applying a smoothing filter process to a source image may be used. In this case, the image quality improvement engine may generate a high-quality image with reduced random noise from the input image. In addition, an image obtained by applying a grayscale conversion process to a source image may also be used as output data of training data. In this case, the image quality improvement engine may generate a high-quality image with enhanced contrast from the input image. Note that regarding a method for generating paired input data and output data of training data, a method similar to the case of using a superimposed image as training data may be performed.

[0158] Note that the input data of the training data may be an image obtained from an image pickup device having the same image quality tendency as the image pickup device 10. Furthermore, the output data of the training data may be a high-quality image obtained by a high-cost process such as a process using a successive approximation method, or may be a high-quality image obtained by photographing a subject corresponding to the input data using an image pickup device having a higher performance than the image pickup device 10. Furthermore, the output data may be a high-quality image obtained by performing a rule-based noise reduction process. Here, the noise reduction process may include, for example, a process of replacing a high-brightness pixel that is obviously noise appearing in a low-brightness area, which is only one pixel, with an average value of adjacent low-brightness pixel values. Therefore, as learning data, the image quality improvement engine may adopt an image photographed by an image pickup device having a higher performance than the image pickup device used to photograph the input image, or an image obtained by an image pickup step involving a greater number of steps than the image pickup step used to obtain the input image. For example, in the case of using a motion contrast frontal image as an input image, the image quality improvement engine can use the following images as learning data: images obtained by OCTA imaging performed by an OCT imaging device having higher performance than the OCTA imaging device used for OCTA imaging of the input image, or images obtained by an OCTA imaging step involving a greater number of steps than the OCTA imaging step of the input image.

[0159] Note that, although omitted in the description of the present embodiment, a high-quality image generated from a plurality of images to be used as output data for training data can be generated from a plurality of images that have been aligned. As alignment processing, for example, the following processing can be adopted: one of the plurality of images is selected as a template, and then the similarity with other images is determined while changing the position and angle of the template, the displacement amount of each image relative to the template is determined, and each image is corrected based on the displacement amount. In addition, any other existing alignment processing can also be performed.

[0160] Note that in the case of aligning a three-dimensional image, the alignment of the three-dimensional image can be performed by decomposing the three-dimensional image into a plurality of two-dimensional images, then aligning the respective two-dimensional images, and integrating the aligned two-dimensional images. In addition, the alignment of the two-dimensional image can be performed by decomposing the two-dimensional image into one-dimensional images, then aligning the respective one-dimensional images, and integrating the aligned one-dimensional images. Note that these alignment processes can be performed on data used to generate the image instead of the image.

[0161] Furthermore, in the present embodiment, if the image quality improvement possibility determination unit 403 determines that the input image can be processed by the image quality improvement unit 404, the process proceeds to step S540, and the image quality improvement process by the image quality improvement unit 404 is started. In this regard, it is also possible to adopt a configuration in which the output unit 405 causes the determination result of the image quality improvement possibility determination unit 403 to be displayed on the display unit 20, and the image quality improvement unit 404 starts the image quality improvement process in response to an instruction from the examiner. At this time, the output unit 405 may also cause the input image or the imaging condition such as the imaging part obtained for the input image to be displayed on the display unit 20 together with the determination result. In this case, since the image quality improvement process is performed after the examiner has determined whether the result determined by the image quality improvement possibility determination unit 403 is correct, the image quality improvement process based on erroneous determination can be reduced.

[0162] In addition, the following structure can also be adopted: instead of determining by the image quality improvement possibility determination unit 403, the output unit 405 displays the input image or the imaging conditions such as the imaging part obtained for the input image on the display unit 20, and the image quality improvement unit 404 starts the image quality improvement processing in response to instructions from the examiner.

[0163] <Second Embodiment>

[0164] Next, refer to Figure 4 and Figure 7An image processing apparatus according to a second embodiment is described. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in the present embodiment, the image quality improvement unit includes a plurality of image quality improvement engines that perform machine learning using training data different from each other and generate a plurality of high-quality images for an input image.

[0165] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0166] The image quality improvement unit 404 according to the present embodiment includes two or more image quality improvement engines that perform machine learning using training data that are different from each other. A method for creating a training data set according to the present embodiment will now be described. Specifically, first, various imaging sites are imaged, and paired groups consisting of paired source images as input data and superimposed images as output data are prepared. Next, the paired groups of each imaging site are grouped to create a training data set. For example, a training data set is created by creating first training data consisting of paired groups obtained by imaging a first imaging site, and creating second training data consisting of paired groups obtained by imaging a second imaging site.

[0167] Thereafter, each image quality improvement engine is made to perform machine learning using each training data. For example, an image quality improvement engine group is prepared, the image quality improvement engine group including a first image quality improvement engine corresponding to a machine learning model trained using the first training data and a second image quality improvement engine corresponding to a machine learning model trained using the second training data.

[0168] Because different training data is used to train the respective machine learning models corresponding to each of these image quality improvement engines, the degree to which each of these image quality improvement engines can improve the image quality of the input image input to the image quality improvement engine will differ depending on the imaging conditions of the input image. Specifically, in the case of the first image quality improvement engine, the degree of improvement in image quality for the input image obtained by imaging the first imaging part is high, and the degree of improvement in image quality for the input image obtained by imaging the second imaging part is low. Similarly, in the case of the second image quality improvement engine, the degree of improvement in image quality for the input image obtained by imaging the second imaging part is high, and the degree of improvement in image quality for the input image obtained by imaging the first imaging part is low.

[0169] Since each set of training data is composed of paired groups grouped according to the imaging part, the images in the image group constituting the paired group have similar image quality trends. Therefore, if the imaging part corresponds to the relevant image quality improvement engine, the image quality improvement engine can perform image quality improvement more effectively than the image quality improvement engine according to the first embodiment. Note that the imaging condition used to group the paired training data is not limited to the imaging part, and may be the imaging angle of view or resolution of the image, or a combination of two or more of these conditions.

[0170] The following will refer to Figure 7 A series of image processing operations according to the present embodiment is described. Figure 7 is a flowchart of a series of image processing operations according to the present embodiment. Note that the processing in step S710 and step S720 is the same as the processing in step S510 and step S520 according to the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally, after the processing in step S720 is performed, the processing in step S730 can be omitted, and the processing can proceed to step S740.

[0171] After obtaining the imaging conditions of the input image in step S720, the process proceeds to step S730. In step S730, the image quality improvement possibility determination unit 403 uses the imaging condition group obtained in step S720 to determine whether any one of the image quality improvement engine groups included in the image quality improvement unit 404 can process the input image.

[0172] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engine group can process the input image, the process proceeds to step S760. On the other hand, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can process the input image, the process proceeds to step S740. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, similar to the first embodiment, even if it is determined that the image quality improvement engine cannot process certain imaging conditions, the process in step S740 may be performed.

[0173] In step S740, the image quality improvement unit 404 selects an image quality improvement engine from the image quality improvement engine group to perform image quality improvement processing based on the imaging conditions of the input image obtained in step S720 and information related to the training data of the image quality improvement engine group. Specifically, for example, the image quality improvement unit 404 selects an image quality improvement engine that has information of training data related to the same imaging part or surrounding imaging parts for the imaging part in the imaging condition group obtained in step S720 and is capable of performing image quality improvement with a high degree of image quality improvement. In the above example, if the imaging part is the first imaging part, the image quality improvement unit 404 selects the first image quality improvement engine.

[0174] In step S750, the image quality improvement unit 404 generates a high-quality image generated by improving the image quality of the input image using the image quality improvement engine selected in step S740. Thereafter, in step S760, if a high-quality image is generated in step S750, the output unit 405 outputs the high-quality image and causes the display unit 20 to display the high-quality image. On the other hand, if it is determined in step S730 that the image quality improvement process is impossible, the output unit 405 outputs the input image and causes the display unit 20 to display the input image. Note that when causing the display unit 20 to display the high-quality image, the output unit 405 may also cause the display unit 20 to display information indicating that the high-quality image is a high-quality image generated using the image quality improvement engine selected by the image quality improvement unit 404.

[0175] As described above, the image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines that learn using learning data that are different from each other. Here, each of the plurality of image quality improvement engines learns using learning data that are different from each other regarding at least one of the imaging conditions of the imaging part, the imaging angle of view, the frontal image at different depths, and the image resolution. The image quality improvement unit 404 generates a high-quality image using the image quality improvement engine according to at least one of the imaging conditions of the imaging part, the imaging angle of view, the frontal image at different depths, and the image resolution of the input image.

[0176] With this configuration, the image processing apparatus 400 according to the present embodiment can generate a high-quality image more effectively.

[0177] Although in the present embodiment, the image quality improvement unit 404 selects the image quality improvement engine to be used for the image quality improvement process based on the imaging condition of the input image, the process for selecting the image quality improvement engine is not limited thereto. For example, the output unit 405 may cause the imaging condition of the input image obtained and the image quality improvement engine group to be displayed on the user interface of the display unit 20, and the image quality improvement unit 404 may select the image quality improvement engine to be used for the image quality improvement process according to the instruction from the examiner. Note that the output unit 405 may cause information related to the training data used for learning by each image quality improvement engine to be displayed on the display unit 20 together with the image quality improvement engine group. Note that the information related to the training data used for learning by the image quality improvement engine may be displayed in any form, and, for example, the image quality improvement engine group may be displayed using a name associated with the training data used for learning.

[0178] In addition, the output unit 405 may cause the image quality improvement engine selected by the image quality improvement unit 404 to be displayed on the user interface of the display unit 20, and may accept an instruction from the examiner. In this case, the image quality improvement unit 404 may determine whether to finally select the relevant image quality improvement engine as the image quality improvement engine to be used for the image quality improvement process according to the instruction from the examiner.

[0179] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data for the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0180] <Third Embodiment>

[0181] Next, refer to Figure 4 and Figure 7 An image processing apparatus according to a third embodiment is described. In the first and second embodiments, the imaging condition obtaining unit 402 obtains an imaging condition group from the data structure of an input image, etc. In contrast, in the present embodiment, the imaging condition obtaining unit estimates an imaging part or imaging area of ​​the input image using an imaging position estimation engine based on the input image.

[0182] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the second embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the second embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first and second embodiments, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first and second embodiments, and description of these components will be omitted hereinafter.

[0183] The imaging condition obtaining unit 402 according to the present embodiment includes an imaging position estimation engine for estimating an imaging part or imaging area present in the input image obtained by the obtaining unit 401. According to the technology for estimating the imaging position of the imaging position estimation engine of the present embodiment, a machine learning algorithm is used for performing estimation processing.

[0184] In the present embodiment, training data consisting of a paired group consisting of input data as an image and output data as an imaging site label or an imaging area label corresponding to the input data is used to train a machine learning model related to an imaging position estimation technique using a machine learning algorithm. Here, the term "input data" refers to an image (input image) having a specific imaging condition assumed for a processing object. An image obtained from an imaging device having the same image quality tendency as the imaging device 10 can be used as input data, and it is better if the imaging device is a device of the same model as the imaging device 10 and is provided with the same settings as the imaging device 10. The type of the imaging site label or the imaging area label used as the output data may be a related imaging site or imaging area at least partially included in the input data. For example, in the case of OCT, the types of the imaging site label used as the output data may be "macular area", "optic nerve head", "macular area and optic nerve head", and "others".

[0185] By learning using such training data, the imaging position estimation engine according to the present embodiment can output information indicating the position of the imaging part or imaging area presented in the input image. In addition, for each imaging part label or imaging area label of the required level of detail, the imaging position estimation engine can also output the probability of becoming a relevant imaging part or imaging area. By using the imaging position estimation engine, the imaging condition acquisition unit 402 can estimate the imaging part or imaging area of ​​the input image based on the input image, thereby obtaining the imaging part or imaging area as the imaging condition for the input image. Please note that in the case where the imaging position estimation engine outputs the probability of becoming a relevant imaging part or imaging area for each imaging part label or imaging area label, the imaging condition acquisition unit 402 obtains the imaging part or imaging area with the highest probability as the imaging condition for the input image.

[0186] Next, similar to the second embodiment, referring to Figure 7 The flowchart in describes a series of image processing operations according to the present embodiment. Note that since the processing of step S710 and steps S730 to S760 according to the present embodiment is the same as the processing in these steps in the second embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally, after the processing in step S720, the processing in step S730 may be omitted, and the processing may proceed to step S740.

[0187] When the input image is obtained in step S710, the process proceeds to step S720. In step S720, the imaging condition obtaining unit 402 obtains an imaging condition group of the input image obtained in step S710.

[0188] Specifically, the imaging condition group stored in the data structure constituting the input image is obtained according to the data format of the input image. In addition, if the information related to the imaging part or the imaging area is not included in the imaging condition group, the imaging condition acquisition unit 402 inputs the input image to the imaging position estimation engine to estimate which imaging part is to be imaged to obtain the input image. Specifically, the imaging condition acquisition unit 402 inputs the input image to the imaging position estimation engine, evaluates the probability of each imaging part label output for the imaging part label group, and sets and obtains the imaging part with the highest probability as the imaging condition of the input image.

[0189] Note that when imaging conditions other than the imaging part or imaging area are not stored in the input image, the imaging condition obtaining unit 402 may obtain an imaging information group including an imaging condition group from the imaging device 10 or an image management system (not shown).

[0190] The subsequent processing is the same as the series of image processing operations according to the second embodiment, and thus a description thereof is omitted here.

[0191] As described above, the imaging condition obtaining unit 402 according to the present embodiment functions as an estimation unit that estimates at least one of an imaging part and an imaging area of ​​an input image. The imaging condition obtaining unit 402 includes an imaging position estimation engine that performs learning using images each having a label of an imaging part or an imaging area attached thereto as learning data, and when an input image is input to the imaging position estimation engine, the imaging position estimation engine estimates the imaging part or the imaging area of ​​the input image.

[0192] Therefore, the image processing device 400 according to the present embodiment can obtain the imaging conditions regarding the imaging part or imaging area of ​​the input image based on the input image.

[0193] Note that in the present embodiment, in the case where the imaging condition group does not include information about the imaging part or the imaging area, the imaging condition obtaining unit 402 uses the imaging position estimation engine to perform estimation on the imaging part or the imaging area of ​​the input image. However, the case where the imaging position estimation engine is used to perform estimation on the imaging part or the imaging area is not limited to this case. In the case where the information about the imaging part or the imaging area included in the data structure of the input image is not sufficient as information of the required level of detail, the imaging condition obtaining unit 402 may also use the imaging position estimation engine to perform estimation on the imaging part or the imaging area.

[0194] Furthermore, regardless of whether the data structure of the input image includes information about the imaging part or the imaging area, the imaging condition obtaining unit 402 can estimate the imaging part or the imaging area of ​​the input image using the imaging position estimation engine. In this case, the output unit 405 can cause the display unit 20 to display the estimation result output from the imaging position estimation engine and the information about the imaging part or the imaging area included in the data structure of the input image, and the imaging condition obtaining unit 402 can make determinations about these imaging conditions according to the instructions of the examiner.

[0195] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0196] <Fourth Embodiment>

[0197] Next, refer to Figure 4 , Figure 5 , Figure 8 and Fig. 9 An image processing device according to a fourth embodiment is described. In this embodiment, the image quality improvement unit enlarges or reduces the input image so that the size of the input image becomes an image size that can be processed by the image quality improvement engine. In addition, the image quality improvement unit generates a high-quality image by reducing or enlarging the output image from the image quality improvement engine so that the image size of the output image becomes the image size of the input image.

[0198] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0199] The image quality improvement unit 404 according to the present embodiment includes an image quality improvement engine similar to the image quality improvement engine according to the first embodiment. However, in the present embodiment, a pairing group of input data and output data consisting of an image group in which each image of the input data and each image of the output data are enlarged or reduced to a certain image size is used as training data for learning by the image quality improvement engine.

[0200] Now refer to Figure 8 The training data of the image quality improvement engine according to the present embodiment is described. Figure 8 As shown, for example, let us consider the following case: there are low-quality images Im810 and high-quality images Im820 that are smaller than a certain image size set for training data. In this case, the low-quality image Im810 and the high-quality image Im820 are enlarged respectively to make them a certain image size set for training data. Then the enlarged low-quality image Im811 and the enlarged high-quality image Im821 are taken as a pair, and the relevant pair is used as a piece of training data.

[0201] Note that, similar to the first embodiment, for the input data of the training data, an image (input image) having specific imaging conditions assumed to be the processing object is used, and the relevant specific imaging conditions are a predetermined imaging part, an imaging system, and an imaging angle. In other words, unlike the first embodiment, according to the present embodiment, the image size is not included in the specific imaging conditions.

[0202] The image quality improvement unit 404 according to the present embodiment generates a high-quality image by improving the image quality of an input image using an image quality improvement engine that learns using such training data. At this time, the image quality improvement unit 404 generates a transformed image by enlarging or reducing the input image so that it becomes a certain image size set for the training data, and inputs the transformed image to the image quality improvement engine. In addition, the image quality improvement unit 404 generates a high-quality image by reducing or enlarging the output image from the image quality improvement engine so that it becomes the image size of the input image. Therefore, even in the case where the input image has an image size that cannot be processed according to the first embodiment, the image quality improvement unit 404 according to the present embodiment can generate a high-quality image by improving the image quality of the input image by the image quality improvement engine.

[0203] Next, refer to Figure 5 and Fig. 9 A series of image processing operations according to the present embodiment is described. Fig. 9is a flowchart showing the image quality improvement processing according to the present embodiment. Note that the processing in step S510, step S520, and step S550 according to the present embodiment is the same as the processing in these steps in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally for the imaging conditions other than the image size, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0204] In step S520, similar to the first embodiment, when the imaging condition acquisition unit 402 obtains the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether the image quality improvement engine included in the image quality improvement unit 404 can process the input image. Specifically, regarding the imaging condition of the input image, the image quality improvement possibility determination unit 403 determines whether the image quality improvement engine can process the imaging part, the imaging system, and the imaging angle of view. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not perform determination regarding the image size.

[0205] The image quality improvement possibility determination unit 403 performs determination regarding the imaging part, the imaging system, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S540. On the other hand, in the case where the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot process the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, even if it is determined that the input image cannot be processed based on one or more conditions of the imaging part, the imaging system, and the imaging angle of view, the image quality improvement process in step S540 may be performed.

[0206] When the process enters step S540, Fig. 9 In the image quality improvement process according to the present embodiment, first, in step S910, the image quality improvement unit 404 enlarges or reduces the input image to a certain image size set for training data, thereby generating a transformed image.

[0207] Next, in step S920 , the image quality improving unit 404 inputs the generated transformed image to the image quality improving engine to obtain a transformed image with high image quality after image quality improvement.

[0208] Thereafter, in step S930, the image quality improvement unit 404 reduces or enlarges the transformed image having high image quality to the image size of the input image to generate a high-quality image. When the image quality improvement unit 404 generates a high-quality image in step S930, the image quality improvement process according to the present embodiment ends, and the process proceeds to step S550. Since the process in step S550 is the same as that in step S550 of the first embodiment, its description is omitted here.

[0209] As described above, the image quality improvement unit 404 according to the present embodiment adjusts the image size of the input image to an image size that can be processed by the image quality improvement engine, and inputs the resized resulting image into the image quality improvement engine. The image quality improvement unit 404 generates a high-quality image by adjusting the image size of the output image from the image quality improvement engine to the original image size of the input image. Therefore, the image processing device 400 of the present embodiment can also use the image quality improvement engine to improve the image quality of the input image having an image size that cannot be processed according to the first embodiment, thereby generating a high-quality image suitable for image diagnosis.

[0210] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0211] <Fifth Embodiment>

[0212] Next, refer to Figure 4 , Figure 5 , Fig.10 and Fig.11 An image processing apparatus according to a fifth embodiment is described. In this embodiment, an image quality improving unit generates a high-quality image through image quality improving processing based on a certain resolution performed by an image quality improving engine.

[0213] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0214] The image quality improvement unit 404 according to the present embodiment includes an image quality improvement engine similar to that of the first embodiment. However, in the present embodiment, the training data used for learning by the image quality improvement engine is different from the training data in the first embodiment. Specifically, after an image group consisting of a pairing group of input data and output data of the training data is enlarged or reduced to an image size such that the resolution of the image group becomes an image size of a certain resolution, padding is performed so that the image size of each image of the image group becomes a certain image size that is large enough. Here, the phrase "resolution of the image group" refers to, for example, the spatial resolution of the camera device or the resolution with respect to the camera area.

[0215] Now refer to Fig.10 The training data of the image quality improvement engine according to the present embodiment is described. Fig.10 As shown, for example, let us consider the following case: there are a low-quality image Im1010 and a high-quality image Im1020 having a resolution lower than a certain resolution set for the training data. In this case, the low-quality image Im1010 and the high-quality image Im1020 are enlarged respectively so that the resolution becomes the certain resolution set for the training data. In addition, the enlarged low-quality image Im1010 and the high-quality image Im1020 are padded respectively so as to become a certain image size set for the training data. Then, the enlarged and padded low-quality image Im1011 and the high-quality image Im1021 are taken as a pair, and the pair is used as a piece of training data.

[0216] Note that the phrase "certain image size set for training data" means that the image (input image) assumed to be the processing object can become the maximum image size such that the resolution of the image becomes a certain resolution when enlarged or reduced. In the case where the certain image size is not large enough, there is a possibility that when the image input to the image quality improvement engine is enlarged, the image will become an image size that cannot be processed by the machine learning model.

[0217] In addition, according to the characteristics of the machine learning model, the filled area is filled with a fixed pixel value, filled with a neighboring pixel value, or filled with a mirror image, so that image quality improvement can be effectively performed. Note that similar to the first embodiment, for input data, an image with a specific imaging condition assumed to be a processing object is used, and the specific imaging condition in question is a predetermined imaging part, imaging system, and imaging angle of view. In other words, unlike the first embodiment, according to the present embodiment, the image size is not included in the specific imaging condition.

[0218] The image quality improvement unit 404 according to the present embodiment generates a high-quality image by improving the image quality of the input image using the image quality improvement engine that learns using such training data. At this time, the image quality improvement unit 404 generates a transformed image by enlarging or reducing the input image to become a certain resolution set for the training data. In addition, the image quality improvement unit 404 pads the transformed image so that the transformed image becomes a certain image size set for the training data, thereby generating a padded image, and inputs the padded image to the image quality improvement engine.

[0219] In addition, for the padded image with high image quality output from the image quality improvement engine, the image quality improvement unit 404 trims only the area corresponding to the area to be padded, thereby generating a transformed image with high image quality. Thereafter, the image quality improvement unit 404 reduces or enlarges the generated transformed image with high image quality to the image size of the input image, thereby generating a high-quality image.

[0220] Therefore, even in the case where the input image has an image size that cannot be processed according to the first embodiment, the image quality improving unit 404 according to the present embodiment can generate a high-quality image by improving the image quality of the input image by the image quality improving engine.

[0221] Next, refer to Figure 5 and Fig.11 A series of image processing operations according to the present embodiment is described. Fig.11 is a flowchart showing the image quality improvement processing according to the present embodiment. Note that the processing in step S510, step S520, and step S550 according to the present embodiment is the same as the processing in these steps in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally for the imaging conditions other than the image size, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0222] In step S520, similar to the first embodiment, when the imaging condition acquisition unit 402 obtains the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether the image quality improvement engine included in the image quality improvement unit 404 can process the input image. Specifically, regarding the imaging condition of the input image, the image quality improvement possibility determination unit 403 determines whether the image quality improvement engine can process the imaging part, the imaging system, and the imaging angle of view. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not perform determination regarding the image size.

[0223] The image quality improvement possibility determination unit 403 performs determination regarding the imaging part, the imaging system, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S540. On the other hand, in the case where the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot process the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, even if it is determined that the input image cannot be processed based on one or more of the imaging part, the imaging system, and the imaging angle of view, the image quality improvement process in step S540 may be performed.

[0224] When the process enters step S540, Fig.11 In the image quality improvement process according to the present embodiment, first, in step S1110, the image quality improvement unit 404 enlarges or reduces the input image to a certain resolution set for the training data, thereby generating a transformed image.

[0225] Next, in step S1120, the image quality improvement unit 404 fills the generated transformed image so that the transformed image becomes the image size set for the training data, thereby generating a filled image. At this time, for the area to be filled, the image quality improvement unit 404 fills the area by using fixed pixel values ​​or using adjacent pixel values ​​or by mirror filling according to the characteristics of the machine learning model, thereby effectively improving the image quality.

[0226] In step S1130 , the image quality improving unit 404 inputs the filling image to the image quality improving engine, thereby obtaining a filling image with high image quality after image quality improvement.

[0227] Next, in step S1140 , for the padded image with high image quality, the image quality improving unit 404 trims only the region corresponding to the region padded in step S1120 , thereby generating a converted image with high image quality.

[0228] Thereafter, in step S1150, the image quality improvement unit 404 reduces or enlarges the transformed image having high image quality to the image size of the input image to generate a high-quality image. When the image quality improvement unit 404 generates a high-quality image in step S1130, the image quality improvement processing according to the present embodiment ends, and the processing proceeds to step S550. Since the processing in step S550 is the same as that in step S550 of the first embodiment, the description of the processing is omitted here.

[0229] As described above, the image quality improvement unit 404 according to the present embodiment adjusts the image size of the input image so that the resolution of the input image becomes a predetermined resolution. In addition, for the input image whose image size is adjusted, the image quality improvement unit 404 generates a padded image that is padded so that the adjusted image size becomes an image size that can be processed by the image quality improvement engine, and then inputs the padded image to the image quality improvement engine. Thereafter, the image quality improvement unit 404 trims the output image from the image quality improvement engine so as to trim only the area corresponding to the area to be padded. Then, the image quality improvement unit 404 adjusts the image size of the trimmed image to the original image size of the input image, thereby generating a high-quality image.

[0230] Therefore, even in the case where the input image has an image size that cannot be processed according to the first embodiment, the image quality improvement unit 404 of the present embodiment can improve the image quality of the input image by the image quality improvement engine, thereby generating a high-quality image. In addition, by using the image quality improvement engine that learns using training data based on resolution, in some cases, the image quality of the input image can be improved more effectively than in the case of the image quality improvement engine according to the fourth embodiment that processes only images of the same image size.

[0231] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0232] <Sixth Embodiment>

[0233] Next, refer to Figure 4 , Figure 5 , Fig.12 and Fig.13 An image processing apparatus according to a sixth embodiment is described. In this embodiment, an image quality improvement unit generates a high-quality image by performing image quality improvement processing on each region of a certain image size of an input image.

[0234] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0235] The image quality improvement unit 404 according to the present embodiment includes an image quality improvement engine similar to that of the first embodiment. However, in the present embodiment, the training data used for learning by the image quality improvement engine is different from the training data in the first embodiment. Specifically, a pairing group of input data as a low-quality image and output data as a high-quality image constituting the training data is composed of rectangular area images of a certain image size corresponding to the positional relationship in the low-quality image and the high-quality image. Note that the rectangular area is an example of a partial area, and the partial area does not need to be rectangular and can be any shape.

[0236] Now refer to Fig.12 The training data of the image quality improvement engine according to the present embodiment is described. Fig.12 As shown, the following case will be considered: for example, in one pairing group constituting training data, there is a source image Im1210 as a low-quality image and a superimposed image Im1220 as a high-quality image. In this case, in the first embodiment, the source image Im1210 is used as input data, and the superimposed image Im1220 is used as output data of the training data.

[0237] In contrast, in the present embodiment, the rectangular area image R1211 in the source image Im1210 is used as input data, and the rectangular area image R1221 of the same imaging area as the rectangular area image R1211 in the superimposed image Im1220 is used as output data. In addition, a pair of training data (hereinafter referred to as the "first rectangular area image pair") is composed of the rectangular area image R1211 as input data and the rectangular area image R1221 as output data. Here, it is assumed that the rectangular area image R1211 and the rectangular area image R1221 are images of a certain image size. Note that the source image Im1210 and the superimposed image Im1220 can be aligned by any method. In addition, the positional relationship corresponding to the rectangular area image R1211 and the rectangular area image R1221 can be identified by any method such as template matching. Note that, depending on the design of the image quality improvement engine, the image sizes and dimensions of the input data and the output data may be different from each other. For example, in the case where the processing target is an OCT image, when the input data is a part of a B-scan image (two-dimensional image), the output data may be a part of an A-scan image (one-dimensional image).

[0238] The certain image size related to the rectangular area images R1211 and R1221 can be determined, for example, based on the common divisor of the group of pixel numbers of each dimension corresponding to the image size group of the image (input image) assumed to be the processing object. In this case, the positional relationship between a group of rectangular area images output by the image quality improvement engine can be prevented from overlapping. Specifically, let us consider the following case: for example, the image assumed to be the processing object is a two-dimensional image, the first image size in the image size group is a width of 500 pixels and a height of 500 pixels, and the second image size in the image size group is a width of 100 pixels and a height of 100 pixels. Here, the certain image size related to the rectangular area images R1211 and R1221 is selected from the common divisor of each side. In this case, for example, the certain image size is selected from a width of 100 pixels and a height of 100 pixels, a width of 50 pixels and a height of 50 pixels, or a width of 25 pixels and a height of 25 pixels, etc.

[0239] Under the assumption that the image to be processed has three dimensions, the number of pixels related to width, height, and depth is determined. Note that a plurality of rectangular areas can be set for a pair of a low-quality image corresponding to input data and a high-quality image corresponding to output data. Thus, for example, a rectangular area image R1212 in the source image Im1210 is used as input data, and a rectangular area image R1222 of the same imaging area as the rectangular area image R1212 in the superimposed image Im1220 is used as output data. In addition, a pair of training data is composed of the rectangular area image R1212 as input data and the rectangular area image R1222 as output data. In this way, a rectangular area image pair different from the first rectangular area image pair can be created.

[0240] Note that by creating a large number of rectangular area image pairs while changing the image of the rectangular area into images with different coordinates, the content of the pairing groups constituting the training data can be enhanced, and it can be expected that effective image quality improvement will be performed by the image quality improvement engine that uses the relevant training pairs for learning. However, a construction can be adopted so that pairings that do not contribute to the image quality improvement performed by the machine learning model are not added to the training data. For example, if the image quality of a rectangular area image created from a high-quality image as output data included in one pairing is not suitable for diagnosis, there is a possibility that the image output by the image quality improvement engine that uses such training data for learning will also have an image quality that is not suitable for image diagnosis. Therefore, the pairing including such a high-quality image can be removed from the training data.

[0241] Furthermore, for example, in the case where there is a large difference in average brightness or brightness distribution between the two rectangular area images forming a pair and the rectangular area image created from the low-quality image and the rectangular area image created from the high-quality image, the pair in question may be removed from the training data. If learning is performed using such training data, there is a possibility that the image quality improvement engine will output an image that is not suitable for image diagnosis and has a brightness distribution that is greatly different from that of the input image.

[0242] In addition, let us consider a case where, for example, between two rectangular area images forming a pair, there is a large difference between the rectangular area image created from the low-quality image and the rectangular area image created from the high-quality image regarding the structure or position of the imaging target to be presented. In this case, there is a possibility that the image quality improvement engine that uses such training data for learning will output an image that is not suitable for image diagnosis in which the structure or position of the imaging target is greatly different from that of the input image. Therefore, such a pair can be removed from the training data.

[0243] Note that, similar to the first embodiment, for the input data of the training data, an image having a specific imaging condition assumed to be a processing object is used, and the relevant specific imaging condition is a predetermined imaging part, an imaging system, and an imaging angle. In other words, unlike the first embodiment, according to the present embodiment, the image size is not included in the specific imaging condition.

[0244] The image quality improvement unit 404 according to the present embodiment generates a high-quality image by improving the image quality of the input image using an image quality improvement engine that learns using such training data. At this time, the image quality improvement unit 404 divides the input image continuously and seamlessly into a group of rectangular area images having a certain image size set for the training data. The image quality improvement unit 404 improves the image quality of each image in the rectangular area image group into which the input image is divided by the image quality improvement engine, thereby generating a group of rectangular area images with high image quality. Thereafter, the image quality improvement unit 404 arranges the generated group of rectangular area images with high image quality according to the positional relationship between the rectangular area images in the input image, and combines the rectangular area images, thereby generating a high-quality image. Here, when learning, if there is a corresponding positional relationship between the input data and the output data that form the image pair, the rectangular area of ​​each input data and the output data can be cut out (extracted) from any position in the low-quality image and the high-quality image. On the other hand, when improving the image quality, the input image can be continuously and seamlessly divided into the rectangular area image group. In addition, the image size of each pair of images during learning and the image size of each rectangular area image during image quality improvement can be set to correspond to each other (for example, to the same size). In this way, it is ensured that while improving learning efficiency, the problem of not being able to obtain an image when insufficient parts or unnecessary calculations occur will not occur.

[0245] Therefore, by improving the image quality of the input image in units of rectangular areas and combining images with improved image quality, the image quality improvement unit 404 of this embodiment can also generate a high-quality image by improving the image quality of images of an image size that cannot be processed according to the first embodiment.

[0246] Next, refer to Figure 5 , Fig.13 and FIG. 14A to FIG. 14D A series of image processing operations according to the present embodiment is described. Fig.13is a flowchart showing the image quality improvement processing according to the present embodiment. Note that the processing in step S510, step S520, and step S550 according to the present embodiment is the same as the processing in these steps in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally for the imaging conditions other than the image size, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0247] In step S520, similar to the first embodiment, when the imaging condition acquisition unit 402 obtains the imaging condition group of the input image, the process proceeds to step S530. In step S530, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether the image quality improvement engine included in the image quality improvement unit 404 can process the input image. Specifically, regarding the imaging condition of the input image, the image quality improvement possibility determination unit 403 determines whether the image quality improvement engine can process the imaging part, the imaging system, and the imaging angle of view. Unlike the first embodiment, the image quality improvement possibility determination unit 403 does not perform determination regarding the image size.

[0248] The image quality improvement possibility determination unit 403 performs determination regarding the imaging part, the imaging system, and the imaging angle of view, and if it is determined that the input image can be processed, the process proceeds to step S540. On the other hand, in the case where the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot process the input image based on these imaging conditions, the process proceeds to step S550. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, even if it is determined that the input image cannot be processed based on one or more of the imaging part, the imaging system, and the imaging angle of view, the image quality improvement process in step S540 may be performed.

[0249] When the process enters step S540, Fig.13 The image quality improvement process according to this embodiment is shown in FIG. FIG. 14A to FIG. 14D In the image quality improvement process according to this embodiment, first, in step S1310, as shown in FIG. Fig.14A As shown, the input image is divided continuously and seamlessly into a set of rectangular area images of a certain image size (the size shown in R1411) set for the training data. Fig.14AAn example of an input image Im1410 divided into a set of rectangular area images R1411 to R1426 of a certain image size is shown. Note that, as described above, depending on the design of the image quality improvement engine, the image input to the image quality improvement engine and the image output from the image quality improvement engine may differ from each other in image size or dimension. In this case, in order to ensure that there is no loss in the combined high-quality image generated in step S1320, the position where the input image is divided can be adjusted by overlapping the position or by separating the position. Fig. 14B An example of overlapping the positions where the input image is divided is shown. Fig. 14B In the figure, reference numerals R1411' and R1412' denote overlapping regions. Although not shown to avoid complicating the drawing, it is assumed that R1413 to R1426 have similar overlapping regions R1413' to R1426'. Note that in Fig. 14B In the case of , the rectangular area size set for the training data is the size shown in R1411'. Because there is no data in the periphery (upper edge, lower edge, left edge and right edge) of the image of the input image Im1410, filling is performed by filling with fixed pixel values, by filling with adjacent pixel values, or by mirror filling. In addition, depending on the image quality improvement engine, in some cases, due to the filtering process, the accuracy of image quality improvement is reduced in the inner periphery (upper edge, lower edge, left edge and right edge) of the image. Therefore, the rectangular area image can be set so that the position where the image is divided is as follows Fig. 14B The rectangular area images may be overlapped as in the example shown, and the rectangular area images may be trimmed at a portion and then combined to obtain the final image. The size of the rectangular area is set according to the characteristics of the image quality improvement engine. Note that although Fig.14A and Fig. 14B The OCT tomography image is used as an example, but Fig. 14C and Fig.14D As shown, the input image (Im1450) may be a frontal image such as an OCTA in-plane image, and similar processing may be performed on it. Note that the size of the rectangular area image is appropriately set according to the type of the image to be processed and the image quality improvement engine.

[0250] Next, in step S1320, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of each rectangular area image in the group of rectangular area images R1411 to R1426, or when setting an overlapping area, improves the image quality of each rectangular area image in the group of rectangular area images R1411' to R1426', thereby generating a group of rectangular area images with high image quality.

[0251] Subsequently, in step S1330, the image quality improvement unit 404 generates a high-quality image by arranging and combining each rectangular area image in the group of rectangular area images with high image quality generated according to the same positional relationship as the positional relationship of the group of rectangular area images R1411 to R1426 obtained by dividing the input image. In the case of setting an overlapping area, the image quality improvement unit 404 generates a high-quality image by cutting out and combining the rectangular area images R1411 to R1426 after arranging the images according to the same positional relationship as the respective rectangular area images R1411' to R1426'. Note that a construction that uses the overlapping area to correct the brightness values ​​of the rectangular area images R1411' to R1426' may be adopted. For example, a rectangular area image to be used as a reference is arbitrarily set. Then, by measuring the brightness values ​​of the same coordinate points in the adjacent rectangular images having an area overlapping with the reference rectangular image, the difference (ratio) between the brightness values ​​of the adjacent images can be found. Similarly, by determining the difference (ratio) between the brightness values ​​at the overlapping areas of all images, correction can be performed to eliminate the unevenness between the overall brightness values. Note that it is not necessary to use the entire overlapping area for brightness value correction, and it is not necessary to use a portion of the overlapping area (the number of pixels on the periphery).

[0252] As described above, the image quality improvement unit 404 according to the present embodiment divides the input image into a plurality of rectangular area images (third images) R1411 to R1426 of a predetermined image size. Thereafter, the image quality improvement unit 404 inputs the divided plurality of rectangular area images R1411 to R1426 to the image quality improvement engine to generate a plurality of fourth images, and then integrates the plurality of fourth images to generate a high-quality image. Note that in the case where the positional relationship between the group of rectangular areas overlaps during integration, the pixel value group of the rectangular area group may be integrated or rewritten.

[0253] Therefore, even in the case where the input image has an image size that cannot be processed according to the first embodiment, the image quality improvement unit 404 of the present embodiment can improve the image quality of the input image by the image quality improvement engine, and thus can generate a high-quality image. In addition, by creating training data from a plurality of images obtained by dividing low-quality images and high-quality images into predetermined image sizes, a large amount of training data can be created from a small amount of images. Therefore, in this case, the number of low-quality images and high-quality images used to create training data can be reduced.

[0254] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0255] <Seventh Embodiment>

[0256] Next, we will refer to Figures 15 to 17 An image processing apparatus according to a seventh embodiment is described. In this embodiment, an image quality evaluation unit selects an image having the highest image quality from among a plurality of high-quality images output by a plurality of image quality improvement engines according to an inspector's instruction.

[0257] Unless otherwise specified, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to this embodiment is described below centering on the differences from the image processing apparatus according to the first embodiment.

[0258] Fig.15 15 is a diagram showing a schematic structure of the image processing device 1500 according to the present embodiment. In addition to the acquisition unit 401, the imaging condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 1500 according to the present embodiment is also provided with an image quality evaluation unit 1506. Note that the image processing device 1500 may be composed of a plurality of devices, each of which is provided with one or more of these components. Here, since the structures regarding the acquisition unit 401, the imaging condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as those of the image processing device according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted below.

[0259] In addition, similar to the image processing device 400 according to the first embodiment, the image processing device 1500 can be connected to the camera 10, the display unit 20, and other devices (not shown) through any circuit or network. In addition, these devices can be connected to any other device through a circuit or network, and can be integrally configured with any other device. Note that although it is assumed in the present embodiment that these devices are devices separated from each other, some or all of these devices may be integrally configured with each other.

[0260] The image quality improvement unit 404 according to the present embodiment includes two or more image quality improvement engines that perform machine learning using training data that are different from each other. A method for creating a training data set according to the present embodiment will now be described. Specifically, first, pairing groups are prepared, each pairing including input data as a low-quality image and output data as a high-quality image obtained by photographing according to various photographing conditions. Next, a training data set is created by grouping the pairing groups according to a combination of arbitrary photographing conditions. For example, a training data set is created, which includes first training data consisting of pairing groups obtained according to a first combination of photographing conditions and second training data consisting of pairing groups obtained according to a second combination of photographing conditions.

[0261] Thereafter, each image quality improvement engine is made to perform machine learning using each training data set. For example, an image quality improvement engine group is prepared, the image quality improvement engine group including a first image quality improvement engine corresponding to a machine learning model trained using the first training data and a second image quality improvement engine corresponding to a machine learning model trained using the first training data.

[0262] Because different training data is used to train the respective machine learning models corresponding to each of these image quality improvement engines, the degree to which each of these image quality improvement engines can improve the image quality of the input image input into the image quality improvement engine will differ depending on the imaging conditions of the input image. Specifically, in the case of the first image quality improvement engine, the degree of improvement in image quality for the input image obtained by imaging using the first combination of imaging conditions is high, and the degree of improvement in image quality for the image obtained by imaging using the second combination of imaging conditions is low. Similarly, in the case of the second image quality improvement engine, the degree of improvement in image quality for the input image obtained by imaging using the second combination of imaging conditions is high, and the degree of improvement in image quality for the image obtained by imaging using the first combination of imaging conditions is low.

[0263] Because each training data set is composed of paired groups grouped according to a combination of imaging conditions, there will be similar image quality trends between the images in the image groups constituting the relevant paired groups. Therefore, if the combination of imaging conditions corresponds to the training data used to train the relevant image quality improvement engine, the relevant image quality improvement engine can perform image quality improvement more effectively than the image improvement engine according to the first embodiment. Please note that the combination of imaging conditions used to group the paired training data can be any combination of imaging conditions, and for example, it can be a combination of two or more imaging conditions in the group consisting of an imaging part, an imaging angle of view, and an image resolution. In addition, similar to the second embodiment, the training data can be grouped based on a single imaging condition.

[0264] In response to the inspector's instruction, the image quality evaluating unit 1506 selects a high-quality image having the highest image quality from among the plurality of high-quality images generated by the image quality improving unit 404 using the plurality of image quality improving engines.

[0265] The output unit 405 may cause the display unit 20 to display the high-quality image selected by the image quality evaluation unit 1506, or may output the high-quality image to other devices. Note that the output unit 405 may cause the display unit 20 to display a plurality of high-quality images generated by the image quality improvement unit 404, and the image quality evaluation unit 1506 may select a high-quality image with the highest image quality according to an instruction from an inspector who has inspected the image on the display unit 20.

[0266] Therefore, the image processing apparatus 1500 can output a high-quality image having the highest image quality from among a plurality of high-quality images generated using a plurality of image quality improvement engines according to the inspector's instruction.

[0267] The following will refer to Fig.16 and Fig.17 A series of image processing operations according to the present embodiment is described. Fig.16 is a flowchart showing a series of image processing operations according to the present embodiment. Note that the processing in step S1610 and step S1620 according to the present embodiment is the same as the processing in step S510 and step S520 in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S1620 is performed, the processing in step S1630 may be omitted, and the processing may proceed to step S1640.

[0268] In step S1620, similarly to the first embodiment, when the imaging condition obtaining unit 402 obtains the imaging condition group of the input image, the process proceeds to step S1630. In step S1630, similarly to the second embodiment, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether any image quality improvement engine included in the image quality improvement unit 404 can process the input image.

[0269] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engine group can process the input image, the process proceeds to step S1660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can process the input image, the process proceeds to step S1640. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, similar to the first embodiment, even if it is determined that some imaging conditions cannot be processed by any image quality improvement engine, the process in step S1640 may be performed.

[0270] In step S1640, the image quality improving unit 404 inputs the input image obtained in step S1610 to each image quality improving engine in the image quality improving engine group, thereby generating a high-quality image group.

[0271] In step S1650, the image quality evaluation unit 1506 selects an image with the highest image quality from the high-quality image group generated in step S1640. Specifically, first, the output unit 405 causes the high-quality image group generated in step S1640 to be displayed on the user interface of the display unit 20.

[0272] An example of the interface in this case is Fig.17 . An input image Im1710 and high-quality images Im1720, Im1730, Im1740, and Im1750 output by the respective image quality improvement engines of the image quality improvement engine group are displayed on the interface. The examiner operates any input device (not shown) to indicate an image with the highest image quality, that is, an image in the image group (high-quality images Im1720 to Im1750) that is most suitable for image diagnosis. Note that since there is also a possibility that an input image that has not undergone image quality improvement by the image quality improvement engine is suitable for image diagnosis, the examiner can add the input image to the image group that is the object of indication.

[0273] Thereafter, the image quality evaluating unit 1506 selects the high-quality image indicated by the checker as the image having the highest image quality.

[0274] In step S1660, the output unit 405 causes the image selected in step S1650 to be displayed on the display unit 20, or outputs the image to other devices. However, if it is determined in step S1630 that it is impossible to process the input image, the output unit 405 outputs the input image as the output image. Note that in the case where the examiner indicates that the input image is selected or in the case where it is impossible to process the input image, the output unit 405 may cause the display unit 20 to display information indicating that the output image is the same as the input image.

[0275] As described above, the image quality improvement unit 404 according to the present embodiment generates a plurality of high-quality images from an input image using a plurality of image quality improvement engines, and the output unit 405 of the image processing apparatus 1500 outputs at least one of the plurality of high-quality images according to the inspector's instruction. In particular, in the present embodiment, the output unit 405 outputs an image having the highest image quality according to the inspector's instruction. In this way, the image processing apparatus 1500 can output a high-quality image having a high image quality from among a plurality of high-quality images generated using a plurality of image quality improvement engines according to the inspector's instruction.

[0276] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 1500. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0277] <Eighth Embodiment>

[0278] Next, we will refer to Fig.15 and Fig.16 An image processing apparatus according to an eighth embodiment is described. In this embodiment, an image quality evaluation unit uses an image quality evaluation engine to select an image having the highest image quality from among a plurality of high-quality images output by a plurality of image quality improvement engines.

[0279] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 1500 according to the seventh embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the seventh embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the seventh embodiment, Fig.15The components of the configuration shown in FIG. 1 are denoted by the same reference numerals as those in the seventh embodiment, and description of these components will be omitted hereinafter.

[0280] The image quality assessment unit 1506 according to the present embodiment includes an image quality assessment engine for assessing the image quality of an input image. The image quality assessment engine outputs an image quality assessment index about the input image. The image quality assessment processing technology for calculating the image quality assessment index in the image quality assessment engine according to the present embodiment uses a machine learning model constructed using a machine learning algorithm. The paired input data constituting the training data for training the machine learning model is an image group consisting of a low-quality image group and a high-quality image group pre-photographed according to various imaging conditions. In addition, the paired output data constituting the training data for training the machine learning model is, for example, an image quality assessment index group, which is set by an examiner performing image diagnosis for each image group of the input data.

[0281] Next, refer to Fig.16 A series of image processing operations according to the present embodiment are described. Note that since the processing in step S1610, step S1620, step S1630, and step S1660 according to the present embodiment is the same as the processing in these steps in the seventh embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S1620 is performed, the processing in step S1630 may be omitted, and the processing may proceed to step S1640.

[0282] In step S1630, similarly to the seventh embodiment, if the image quality improvement possibility determination unit 403 determines that any image quality improvement engine in the image quality improvement engine group can process the input image, the process proceeds to step S1640. Note that, depending on the setting or implementation form of the image processing apparatus 400, similarly to the first embodiment, even if it is determined that some imaging conditions cannot be processed by any image quality improvement engine, the process in step S1640 may be performed.

[0283] In step S1640, the image quality improving unit 404 inputs the input image obtained in step S1610 to each image quality improving engine in the image quality improving engine group, thereby generating a high-quality image group.

[0284] In step S1650, the image quality assessment unit 1506 selects an image with the highest image quality from the high-quality image group generated in step S1640. Specifically, first, the image quality assessment unit 1506 inputs the high-quality image group generated in step S1640 to the image quality assessment engine. The image quality assessment engine calculates an image quality assessment index for each input high-quality image based on learning. The image quality assessment unit 1506 selects a high-quality image for which the highest image quality assessment index among the calculated image quality assessment indexes is calculated. Note that since there is also a possibility that an input image that has not undergone image quality improvement by the image quality improvement engine is suitable for image diagnosis, the image quality assessment unit 1506 may also input the input image into the image quality assessment engine and then add the image quality assessment index of the input image to the selection. Since step S1660 is the same as step S1660 of the seventh embodiment, its description is omitted here.

[0285] As described above, the image processing device 1500 according to the present embodiment further includes an image quality evaluation unit 1506 that evaluates the image quality of a high-quality image. The image quality improvement unit 404 generates a plurality of high-quality images from an input image using a plurality of image quality improvement engines, and the output unit 405 of the image processing device 1500 outputs at least one of the plurality of high-quality images according to the evaluation result of the image quality evaluation unit 1506. In particular, the image quality evaluation unit 1506 according to the present embodiment includes an image quality evaluation engine that uses an evaluation value obtained according to a predetermined evaluation technique as learning data. The image quality evaluation unit 1506 selects a high-quality image of which the evaluation result of the image quality evaluation unit 1506 using the image quality evaluation engine is the highest among the plurality of high-quality images. The output unit 405 outputs the high-quality image having the highest evaluation value selected by the image quality evaluation unit 1506.

[0286] Therefore, the image processing apparatus 1500 according to the present embodiment can easily output a high-quality image most suitable for image diagnosis from among a plurality of high-quality images based on the output of the image quality assessment engine.

[0287] Note that in the present embodiment, the image quality assessment unit 1506 selects a high-quality image having the highest image quality assessment index among the image quality assessment indexes output by the image quality assessment engine, and the output unit 405 causes the display unit 20 to display the selected high-quality image. However, the configuration of the image quality assessment unit 1506 is not limited thereto. For example, the image quality assessment unit 1506 may select a high-quality image having several high image quality assessment indexes among the image quality assessment indexes output by the image quality assessment engine, and the output unit 405 may cause the selected high-quality image to be displayed on the display unit 20. Furthermore, the output unit 405 may cause the image quality assessment index output by the image quality assessment engine to be displayed on the display unit 20 together with the corresponding high-quality image, and the image quality assessment unit 1506 may select the image having the highest image quality according to an instruction from the examiner.

[0288] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 1500. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0289] <Ninth Embodiment>

[0290] Next, refer to Fig.18 and Fig.19 An image processing apparatus according to a ninth embodiment is described. In this embodiment, the authenticity evaluation unit uses an authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in image quality.

[0291] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to this embodiment is described below centering on the differences from the image processing apparatus according to the first embodiment.

[0292] Fig.18The schematic structure of the image processing device 1800 according to the present embodiment is shown. In addition to the acquisition unit 401, the camera condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404 and the output unit 405, the image processing device 1800 according to the present embodiment is also equipped with an authenticity evaluation unit 1807. Note that the image processing device 1800 can be composed of a plurality of devices, each of which is equipped with one or more of these components. Here, since the structures regarding the acquisition unit 401, the camera condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404 and the output unit 405 are the same as those of the image processing device according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted below.

[0293] In addition, similar to the image processing device 400 according to the first embodiment, the image processing device 1800 can be connected to the camera 10, the display unit 20, and other devices (not shown) through any circuit or network. In addition, these devices can be connected to any other device through a circuit or network, and can be integrally configured with any other device. Note that although it is assumed in the present embodiment that these devices are devices separated from each other, some or all of these devices may be integrally configured with each other.

[0294] The authenticity evaluation unit 1807 includes an authenticity evaluation engine. The authenticity evaluation unit 1807 uses the authenticity evaluation engine to evaluate whether the high-quality image generated by the image quality improvement engine has undergone sufficient image quality improvement. The authenticity evaluation processing technology used by the authenticity evaluation engine according to this embodiment uses a machine learning model constructed using a machine learning algorithm.

[0295] The training data used to train the machine learning model includes a pairing group consisting of a high-quality image group that was previously photographed according to various photographing conditions and a label indicating that the relevant image was photographed by a target photographing device (hereinafter referred to as a "true label"). In addition, the training data includes a pairing group consisting of a high-quality image group generated by inputting a low-quality image into an image quality improvement engine with poor image quality improvement accuracy and a label indicating that the relevant image was not photographed by a target photographing device (hereinafter referred to as a "false label").

[0296] Although it is not the case that the authenticity evaluation engine that has been learned using such training data can definitely evaluate whether the input image is obtained by being photographed by a camera, the authenticity evaluation engine can evaluate whether the image looks like an image obtained by being photographed by a camera. Using this feature, by inputting the high-quality image generated by the image quality improvement unit 404 into the authenticity evaluation engine, the authenticity evaluation unit 1807 can evaluate whether the high-quality image generated by the image quality improvement unit 404 has been sufficiently improved in image quality.

[0297] If it is determined by the authenticity evaluation unit 1807 that the high-quality image generated by the image quality improvement unit 404 has undergone sufficient image quality improvement, the output unit 405 causes the display unit 20 to display the high-quality image. On the other hand, if it is determined by the authenticity evaluation unit 1807 that the high-quality image generated by the image quality improvement unit 404 has not undergone sufficient image quality improvement, the output unit 405 causes the display unit 20 to display the input image. Note that when causing the input image to be displayed, the output unit 405 may cause the display unit 20 to display information indicating that the high-quality image generated by the image quality improvement unit 404 has not undergone sufficient image quality improvement or that the displayed image is the input image.

[0298] The following will refer to Fig.19 A series of image processing operations according to the present embodiment is described. Fig.19 is a flowchart showing a series of image processing operations according to the present embodiment. Note that the processing in step S1910 to step S1940 according to the present embodiment is the same as the processing in step S510 to step S540 in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S1920 is performed, the processing in step S1930 may be omitted, and the processing may proceed to step S1940.

[0299] In step S1940, when the image quality improvement unit 404 generates a high-quality image group, the process proceeds to step S1950. In step S1950, the authenticity evaluation unit 1807 inputs the high-quality image generated in step S1940 into the authenticity evaluation engine, and performs authenticity evaluation based on the output of the authenticity evaluation engine. Specifically, in the case where a true label ("true") is output from the authenticity evaluation engine, the authenticity evaluation unit 1807 evaluates that the generated high-quality image has undergone sufficient image quality improvement. On the other hand, in the case where a false label ("false") is output from the authenticity evaluation engine, the authenticity evaluation unit 1807 evaluates that the generated high-quality image has not undergone sufficient image quality improvement.

[0300] In step S1960, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has undergone sufficient image quality improvement, the output unit 405 causes the display unit 20 to display the relevant high-quality image. On the other hand, if the authenticity evaluation unit 1807 determines that the high-quality image generated by the image quality improvement unit 404 has not undergone sufficient image quality improvement, the output unit 405 causes the display unit 20 to display the input image.

[0301] As described above, the image processing apparatus 1800 according to the present embodiment further includes an authenticity evaluation unit 1807 that evaluates the image quality of the high-quality image, and the authenticity evaluation unit 1807 includes an authenticity evaluation engine that evaluates the authenticity of the image. The authenticity evaluation engine includes a machine learning engine for which an image generated by an image quality improvement engine having an image quality improvement processing accuracy lower (poorer) than that of the image quality improvement engine of the image quality improvement unit 404 is used as learning data. In the case where the output of the authenticity evaluation engine of the authenticity evaluation unit indicates that the relevant high-quality image is authentic, the output unit 405 of the image processing apparatus 1800 outputs the high-quality image.

[0302] In this way, according to the image processing apparatus 1800 of the present embodiment, the inspector can effectively confirm that a high-quality image has undergone sufficient image quality improvement.

[0303] Furthermore, the efficiency and accuracy of both the image quality improvement engine and the authenticity assessment engine may be improved by training the machine learning model of the image quality improvement engine and the machine learning model of the authenticity assessment engine in a coordinated manner.

[0304] Note that, although the following configuration is adopted in the present embodiment: the image quality improvement unit 404 generates a single high-quality image, and the authenticity evaluation unit 1807 performs evaluation for the generated single high-quality image, the evaluation performed by the authenticity evaluation unit 1807 is not limited to this. For example, in the case where the image quality improvement unit 404 generates a plurality of high-quality images using a plurality of image quality improvement engines as in the second embodiment, the authenticity evaluation unit 1807 may be configured to perform evaluation for at least one of the plurality of generated high-quality images. In this case, for example, the authenticity evaluation unit 1807 may perform evaluation for all of the plurality of generated high-quality images, or may perform evaluation for only the image indicated by the checker among the plurality of high-quality images.

[0305] In addition, the output unit 405 can cause the display unit 20 to display the determination result of the authenticity evaluation unit 1807 as to whether the high-quality image has undergone sufficient image quality improvement, and can output the high-quality image in the display unit 20 according to the inspector's instruction.

[0306] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other device connected to the image processing device 1800. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0307] <Tenth Embodiment>

[0308] Next, refer to Figure 4 and Figure 5 An image processing apparatus according to a tenth embodiment is described. In this embodiment, an image quality improvement unit divides an input three-dimensional image into a plurality of two-dimensional images, and inputs the plurality of two-dimensional images to an image quality improvement engine, and then combines output images from the image quality improvement engine to generate a three-dimensional high-quality image.

[0309] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0310] The acquisition unit 401 according to the present embodiment acquires a three-dimensional image composed of a set of two-dimensional images that are structurally continuous. Specifically, the three-dimensional image is, for example, a three-dimensional OCT volume image composed of a set of OCT B-scan images (tomographic images). In addition, the three-dimensional image is, for example, a three-dimensional CT volume image composed of a set of axial tomographic images.

[0311] Similar to the first embodiment, the image quality improvement unit 404 includes an image quality improvement engine. Note that the image group of the two-dimensional image constitutes a pairing group of input data and output data as training data of the image quality improvement engine. The image quality improvement unit 404 divides the obtained three-dimensional image into a plurality of two-dimensional images and inputs each two-dimensional image into the image quality improvement engine. In this way, the image quality improvement unit 404 can generate a plurality of two-dimensional high-quality images.

[0312] The output unit 405 combines the plurality of two-dimensional high-quality images generated by the image quality improving unit 404 for the respective two-dimensional images of the three-dimensional image, thereby outputting a three-dimensional high-quality image.

[0313] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that since the processing in steps S510 to S530 and S550 according to the present embodiment is similar to the processing in these steps in the first embodiment, the description of the processing is omitted here. However, in step S510, the acquisition unit 401 obtains a three-dimensional image. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging condition, after the processing in step S520 is performed, the processing in step S530 can be omitted, and the processing can proceed to step S540.

[0314] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be processed by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make the determination in question based on the imaging conditions of the three-dimensional image, or may make the determination based on the imaging conditions related to the plurality of two-dimensional images constituting the three-dimensional image. In step S540, the image quality improvement unit 404 divides the obtained three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided plurality of two-dimensional images to the image quality improvement engine, thereby generating a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated plurality of two-dimensional high-quality images based on the obtained three-dimensional image, thereby generating a three-dimensional high-quality image.

[0315] In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high-quality image. Note that the three-dimensional high-quality image may be displayed in any display form.

[0316] As described above, the image quality improvement unit 404 according to the present embodiment divides the input three-dimensional image into a plurality of two-dimensional images and inputs the plurality of two-dimensional images into the image quality improvement engine. The image quality improvement unit 404 combines the plurality of two-dimensional high-quality images output from the image quality improvement engine to generate a three-dimensional high-quality image.

[0317] Therefore, the image quality improving unit 404 according to the present embodiment can improve the image quality of a three-dimensional image by using an image quality improving engine that performs learning using training data composed of two-dimensional images.

[0318] Note that, similarly to the first embodiment, the output unit 405 may output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similarly to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes may be used: superposition process, MAP estimation process, smoothing filter process, grayscale conversion process, camera using a high-performance camera, high-cost process, and noise reduction process.

[0319] <Eleventh Embodiment>

[0320] Next, refer to Figure 4 and Figure 5 An image processing apparatus according to an eleventh embodiment is described. In this embodiment, an image quality improvement unit divides an input three-dimensional image into a plurality of two-dimensional images, subjects the plurality of two-dimensional images to image quality improvement by a plurality of image quality improvement engines in parallel, and then combines output images from the image quality improvement engines to generate a three-dimensional high-quality image.

[0321] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the tenth embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the tenth embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first and tenth embodiments, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment and the tenth embodiment, and description of these components will be omitted hereinafter.

[0322] The image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines that are the same as those provided in the tenth embodiment. Note that the group of the plurality of image quality improvement engines provided in the image quality improvement unit 404 may be implemented in a manner that enables distributed processing in a group of two or more devices through a circuit or a network, or may be implemented in a single device.

[0323] Similar to the tenth embodiment, the image quality improvement unit 404 divides the obtained three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 uses a plurality of image quality improvement engines to improve the image quality of the plurality of two-dimensional images by sharing the processing therebetween (in parallel), thereby generating a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines a plurality of two-dimensional high-quality images output from a plurality of image quality improvement engines based on the three-dimensional image as a processing object, thereby generating a three-dimensional high-quality image.

[0324] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that since the processing in steps S510 to S530 and S550 according to the present embodiment is the same as that in these steps in the tenth embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0325] In step S530, if the image quality improvement possibility determination unit 403 determines that the input image can be processed by the image quality improvement engine, the process proceeds to step S540. Note that the image quality improvement possibility determination unit 403 may make the determination in question based on the imaging conditions of the three-dimensional image, or may make the determination based on the imaging conditions related to a plurality of two-dimensional images constituting the three-dimensional image.

[0326] In step S540, the image quality improvement unit 404 divides the obtained three-dimensional image into a plurality of two-dimensional images. The image quality improvement unit 404 inputs each of the divided plurality of two-dimensional images into a plurality of image quality improvement engines to perform image quality improvement processing on the two-dimensional images in parallel and generate a plurality of two-dimensional high-quality images. The image quality improvement unit 404 combines the generated plurality of two-dimensional high-quality images based on the obtained three-dimensional image, thereby generating a three-dimensional high-quality image.

[0327] In step S550, the output unit 405 causes the display unit 20 to display the generated three-dimensional high-quality image. Note that the three-dimensional high-quality image may be displayed in any display form.

[0328] As described above, the image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines. The image quality improvement unit 404 divides the input three-dimensional image into a plurality of two-dimensional images, and generates a plurality of two-dimensional high-quality images by using a plurality of image quality improvement engines in parallel. The image quality improvement unit 404 integrates the plurality of two-dimensional high-quality images to generate a three-dimensional high-quality image.

[0329] Therefore, the image quality improvement unit 404 according to the present embodiment can improve the image quality of the three-dimensional image by using the image quality improvement engine, which uses the training data composed of the two-dimensional image for learning. In addition, compared with the tenth embodiment, the image quality of the three-dimensional image can be improved more effectively.

[0330] Note that the training data of the plurality of image quality improvement engines may be different training data according to the processing objects processed by the respective image quality improvement engines. For example, the first image quality improvement engine may use the training data for the first imaging area for learning, and the second image quality improvement engine may use the training data for the second imaging area for learning. In this case, the respective image quality improvement engines may perform image quality improvement more accurately for the two-dimensional image.

[0331] In addition, similar to the first embodiment, the output unit 405 can output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similar to the first embodiment, the output data of the training data for the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes can be used: superposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, camera using a high-performance camera, high-cost processing, and noise reduction processing.

[0332] <Twelfth Embodiment>

[0333] Next, refer to Figure 5 and Fig. 20 An image processing apparatus according to a twelfth embodiment is described. In this embodiment, the obtaining unit 401 obtains an input image from the image management system 2000 instead of the imaging apparatus.

[0334] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus 400 according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0335] Fig. 20 2 is a diagram showing a schematic configuration of an image processing apparatus 400 according to the present embodiment. The image processing apparatus 400 according to the present embodiment is connected to an image management system 2000 and a display unit 20 through any circuit or network. The image management system 2000 is a device and a system configured to receive and store images photographed by any camera or images subjected to image processing. In addition, the image management system 2000 can send images in response to requests from connected devices, perform image processing on stored images, and request other devices to execute requests for image processing. A picture archiving and communication system (PACS) can be included as an example of an image management system.

[0336] The obtaining unit 401 according to the present embodiment may obtain an input image from the image management system 2000 connected to the image processing apparatus 400. Furthermore, the output unit 405 may output a high-quality image generated by the image quality improving unit 404 to the image management system 2000.

[0337] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that since the processing in steps S520 to S540 according to the present embodiment is the same as that in these steps in the first embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0338] In step S510, the obtaining unit 401 obtains, from the image management system 2000 connected to the image processing apparatus 400 via a circuit or a network, an image stored in the image management system 2000 as an input image. Note that the obtaining unit 401 may obtain the input image in response to a request from the image management system 2000. For example, the request may be issued when the image management system 2000 stores an image, or before the stored image is sent to another device, or when the stored image is displayed on the display unit 20. In addition, for example, when the user operates the image management system 2000 to request image quality improvement processing, or when the image analysis function included in the image management system 2000 utilizes a high-quality image, a relevant request may be issued.

[0339] The processing from step S520 to step S540 is the same as that in the first embodiment. When the image quality improvement unit 404 generates a high-quality image in step S540, the processing proceeds to step S550. In step S550, if a high-quality image is generated in step S540, the output unit 405 outputs the high-quality image as an output image to the image management system 2000. If a high-quality image is not generated in step S540, the output unit 405 outputs the above-mentioned input image as an output image to the image management system 2000. Note that, depending on the setting or implementation of the image processing device 400, the output unit 405 can process the output image or convert the data format of the output image so that the image management system 2000 can utilize the output image.

[0340] As described above, the obtaining unit 401 according to the present embodiment obtains the input image from the image management system 2000. Therefore, based on the image stored in the image management system 2000, the image processing device 400 of the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness to the subject or the labor of the person performing the imaging. In addition, the output high-quality image can be stored in the image management system 2000, or can be displayed on the user interface included in the image management system 2000. In addition, the output high-quality image can be utilized by the image analysis function included in the image management system 2000, or the output high-quality image can be sent to other devices connected to the image management system 2000 through the image management system 2000.

[0341] Note that the image processing device 400, the image management system 2000, and the display unit 20 may be connected to other devices (not shown) via a circuit or a network. In addition, although these devices are assumed to be separate devices in this embodiment, some or all of these devices may be integrally formed with each other.

[0342] Furthermore, similar to the first embodiment, the output unit 405 may output the generated high-quality image to the image management system 2000 or other devices connected to the image processing device 400 .

[0343] <Thirteenth Embodiment>

[0344] Next, refer to Figure 4 , Figure 5 , Fig.21A and Fig.21B An image processing apparatus according to a thirteenth embodiment is described. In this embodiment, an image quality improving unit takes a plurality of images as input images to generate a single high-quality image.

[0345] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0346] The obtaining unit 401 according to the present embodiment obtains a plurality of images as processing targets from the imaging device 10 or other devices as input data.

[0347] The image quality improvement unit 404 according to the present embodiment is provided with the same image quality improvement engine as that in the first embodiment. The training data may also be the same as that in the first embodiment. The image quality improvement unit 404 inputs each of the multiple images obtained by the obtaining unit 401 into the image quality improvement engine, and performs a superposition process on the output multiple high-quality images, thereby generating a final high-quality image. Please note that before superimposing the multiple high-quality images, the image quality improvement unit 404 may align the multiple high-quality images by any method.

[0348] The output unit 405 causes the display unit 20 to display the final high-quality image generated by the image quality improving unit 404. Note that the output unit 405 may cause a plurality of input images to be displayed together with the final high-quality image on the display unit 20. Furthermore, the output unit 405 may display the generated plurality of high-quality images together with the final high-quality image or the input images on the display unit 20.

[0349] Next, refer to Figure 5 and Fig.21A A series of image processing operations according to the present embodiment is described. Fig.21A: is a flowchart of the image quality improvement process according to the present embodiment. Note that the processes in step S510 to step S530 according to the present embodiment are similar to those in the first embodiment, and thus description of the processes is omitted here.

[0350] However, in step S510, the obtaining unit 401 obtains a plurality of images, and in steps S520 and S530, the imaging condition is obtained for each of the plurality of images, and it is determined whether each image can be processed by the image quality improvement engine. Note that in the case where the image quality of each input image is to be improved unconditionally for the imaging condition, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540. In addition, in the case where it is determined that the image quality improvement engine cannot process one or more of the plurality of images, the relevant image may be excluded from subsequent processing.

[0351] If the image quality improvement possibility determination unit 403 determines in step S530 that the image quality improvement engine can process a plurality of input images, the process proceeds to step S540. When the process proceeds to step S540, the process starts Fig.21A In the image quality improvement process according to the present embodiment, first, in step S2110, the image quality improvement unit 404 inputs each of a plurality of input images to the image quality improvement engine to generate a high-quality image group.

[0352] Next, in step S2120, the image quality improvement unit 404 performs a superposition process on the generated high-quality image group to generate a final single high-quality image. Note that the superposition process can be performed by an averaging process such as additive averaging or by any other existing process. When superimposing, the image quality improvement unit 404 can align a plurality of high-quality images by any method and then perform superposition. When the image quality improvement unit 404 generates the final high-quality image, the process proceeds to step S550.

[0353] In step S550 , the output unit 405 displays the generated final high-quality image on the display unit 20 .

[0354] As described above, the image quality improvement unit 404 according to the present embodiment generates a single final high-quality image from a plurality of input images. Because the image quality improvement engine performs image quality improvement based on the input images, for example, in the case where a lesion site, etc. is not properly displayed in a specific input image, the lesion site, etc. will be a site with a low pixel value in a high-quality image generated by performing image quality improvement on the relevant input image. On the other hand, in some cases, the lesion site, etc. is properly displayed in other input images photographed at the same position, and the lesion site, etc. will be a site with a high pixel value in a high-quality image generated by performing image quality improvement on the relevant other input images. Therefore, by superimposing these high-quality images, the relevant positions with low or high pixel values ​​can be properly displayed, and a high-quality image with high contrast can be generated. Note that by setting the number of input images to a number smaller than the number of images required for conventional superposition, costs such as the extended shooting time that occurs in conventional technology can be reduced.

[0355] Note that the above-described effect is significant, for example, in the case of using an input image utilizing motion contrast data such as OCTA data.

[0356] Since the motion contrast data is obtained by detecting the temporal change of the imaging target within the time interval in which the imaging is repeatedly performed at the same position of the imaging target, there is a case where only a slight motion can be detected for the motion of the imaging target within a certain time interval, for example. On the other hand, there is a case where the motion of the imaging target can be detected as a large motion when the imaging is performed at different time intervals. Therefore, by superimposing the images generated by subjecting the motion contrast images obtained in each of these cases to image quality improvement, the motion contrast that does not occur or is only slightly detected at a specific timing can be interpolated. Therefore, according to such processing, a motion contrast image with contrast enhancement for a large amount of motion of the imaging target can be generated, and the examiner can determine a more accurate state of the imaging target.

[0357] Therefore, when an image that visualizes a position that changes with time (for example, an OCTA image) is used as an input image, a predetermined part of the subject can be imaged in more detail by superimposing high-quality images obtained at different times.

[0358] Note that, although in the present embodiment, high-quality images are generated from a plurality of input images respectively, and the high-quality images are superimposed to generate a final single high-quality image, the method for generating a single high-quality image from a plurality of input images is not limited to this method. Fig.21BIn the illustrated different example of the image quality improvement process of the present embodiment, when the image quality improvement process is started in step S540, in step S2130, the image quality improvement unit 404 superimposes the input image group to generate a single superimposed input image.

[0359] Thereafter, in step S2140, the image quality improvement unit 404 inputs the single superimposed input image to the image quality improvement engine to generate a single high-quality image. Even when the image quality improvement process is performed in this manner, similar to the aforementioned image quality improvement process, positions with low or high pixel values ​​with respect to a plurality of input images can be appropriately displayed, and a high-contrast high-quality image can be generated. In the case where the above-mentioned motion contrast image (e.g., OCTA image) is used as an input image, this process can also achieve a significant effect.

[0360] Note that in the case of performing the high image quality processing discussed, superimposed images of the same number of input images as the plurality of input images assumed to be the processing object are used as input data for the training data of the image quality improvement engine. Therefore, the image quality improvement engine can perform appropriate image quality improvement processing.

[0361] In addition, regarding the image quality improvement processing according to the present embodiment and the aforementioned different image quality improvement processing, the processing for combining the high-quality image groups or input images is not limited to the superposition processing. For example, a single image may be generated by applying the MAP estimation processing to each of these image groups. In addition, a single image may be generated by combining the high-quality image groups or the input image groups.

[0362] Examples of the case where a single image is generated by combining a high-quality image group or an input image group include the following case: an image having a wide grayscale for a high-brightness area and an image having a wide grayscale for a low-brightness area are used as input images. In this case, for example, an image generated by improving the image quality of an image having a wide grayscale with respect to a high-brightness area is combined with an image generated by improving the image quality of an image having a wide grayscale with respect to a low-brightness area. In this way, an image capable of representing a wider brightness range (dynamic range) can be generated. Note that in this case, an image having a wide grayscale with respect to a high-brightness area and a low-quality image having a wide grayscale with respect to a low-brightness area, which are assumed to be processing objects, can be used as input images of training data for an image quality improvement engine. In addition, a high-quality image corresponding to the input data can be used as output data of training data for an image quality improvement engine.

[0363] In addition, an image with wide grayscale for a high brightness area and an image with wide grayscale for a low brightness area can be combined, and the image quality of the combined image can be improved by the image quality improvement engine. In this case as well, an image capable of representing a wider brightness range can be generated. Please note that in this case, an image assumed to be a processing object generated by combining a low-quality image with wide grayscale for a high brightness area and a low-quality image with wide grayscale for a low brightness area can be used as an input image of training data for the image quality improvement engine. In addition, a high-quality image corresponding to the input data can be used as output data of the training data for the image quality improvement engine.

[0364] In these cases, an image quality improvement engine can be used to improve the image quality of images that can represent a wider brightness range, can be processed with a smaller number of images than in conventional techniques, and can provide images suitable for image analysis at a lower cost.

[0365] Note that any method such as a method of shortening or lengthening the exposure time of the camera may be adopted as a method for capturing an image having a wide grayscale with respect to a high-brightness area and an image having a wide grayscale with respect to a low-brightness area. Furthermore, the method of dividing the grayscale width is not limited to the low-brightness area and the high-brightness area, and the grayscale width may be divided in any manner.

[0366] In addition, in the image quality improvement processing according to the present embodiment, multiple image quality improvement engines can be used to process multiple input images in parallel. Note that, similar to the first embodiment, the output unit 405 can output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similar to the first embodiment, the output data of the training data for the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition processing. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processing can be used: superposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, camera using a high-performance camera device, high-cost processing, and noise reduction processing.

[0367] <Fourteenth Embodiment>

[0368] Next, refer to Figure 4 and Figure 5 An image processing apparatus according to a fourteenth embodiment is described. In this embodiment, an image quality improving unit takes a medium-quality image generated from a plurality of low-quality images as an input image and generates a high-quality image.

[0369] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0370] The obtaining unit 401 according to the present embodiment obtains a medium quality image generated by superimposing a plurality of low quality images from the camera 10 or other devices as input data to be processed. Note that when superimposing low quality images, any alignment processing may be performed.

[0371] The image quality improvement unit 404 according to the present embodiment includes an image quality improvement engine similar to that of the first embodiment. However, the image quality improvement engine of the present embodiment is designed to output a high-quality image when a medium-quality image, which is an image with a medium-level image quality, is input thereto. The medium-quality image is a superimposed image generated by superimposing a plurality of low-quality image groups. In addition, the high-quality image is an image having a higher image quality than the medium-quality image. In addition, with respect to the paired groups constituting the training data for training by the image quality improvement engine, the input data constituting each pairing is a medium-quality image generated in a manner similar to the above-mentioned medium-quality image, and the output data is a high-quality image.

[0372] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404. Note that the output unit 405 may cause the display unit 20 to display the input image together with the high-quality image, and in this case, the output unit 405 may cause the display unit 20 to display information indicating that the input image is an image generated from a plurality of low-quality images.

[0373] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that since the processing in steps S520 to S550 according to the present embodiment is the same as that in the first embodiment, the description of the processing is omitted here.

[0374] In step S510, the obtaining unit 401 obtains a medium-quality image as an input image from the camera 10 or other devices. Note that the obtaining unit 401 can obtain a medium-quality image generated by the camera 10 as an input image in response to a request from the camera 10. For example, such a request can be issued when the camera 10 generates an image, or when the image generated by the camera 10 is displayed on the display unit 20 before being stored in a storage device included in the camera 10, or when the stored image is displayed on the display unit 20 after being stored in the storage device, or when a high-quality image is used for image analysis processing.

[0375] The processing thereafter is the same as that in the first embodiment, so the description of the processing is omitted here.

[0376] As described above, the obtaining unit 401 according to the present embodiment obtains a medium-quality image, which is an image generated using a plurality of images of a predetermined part of a subject as an input image. In this case, since the input image is a clearer image, the image quality improvement engine can generate a high-quality image with higher precision. Note that the number of low-quality images used to generate the medium-quality image may be smaller than the number of images used to generate the conventional superimposed image.

[0377] Note that the medium-quality image is not limited to an image generated by superimposing a plurality of low-quality images, and may be, for example, an image generated by applying MAP estimation processing to a plurality of low-quality images, or may be an image generated by combining a plurality of low-quality images. In the case of combining a plurality of low-quality images, images whose grayscales are different from each other may be combined.

[0378] In addition, similar to the first embodiment, the output unit 405 can output the generated high-quality image to the camera 10 or other devices connected to the image processing device 400. In addition, similar to the first embodiment, the output data of the training data for the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes can be used: superposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, camera using a high-performance camera, high-cost processing, and noise reduction processing.

[0379] <Fifteenth Embodiment>

[0380] Next, refer to Figure 4 and Figure 5An image processing apparatus according to a fifteenth embodiment is described. In this embodiment, an image quality improving unit performs image size enlargement (size increase) with respect to an input image, as well as the processing for image quality improvement according to the first embodiment and the like.

[0381] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0382] The acquisition unit 401 according to the present embodiment obtains an image with a small size (small-size image) as an input image. Please note that the term "small-size image" refers to an image in which the number of pixels constituting the image is less than the number of pixels in an image with a large size (large-size image) output by the image quality improvement engine described later. Specifically, for example, in the case where the size of the large-size image is 1024 pixels in width, 1024 pixels in height, and 1024 pixels in depth, the size of the small-size image is 512 pixels in width, 512 pixels in height, 512 pixels in depth, and the like. In this regard, in the present specification, the term "image size increase" refers to a process of increasing the number of pixels of each image to increase the image size.

[0383] Similar to the first embodiment, the image quality improvement unit 404 according to the present embodiment includes an image quality improvement engine. However, the image quality improvement engine of the present embodiment is configured to increase the image size of the input image while reducing noise and enhancing the contrast of the input image. Therefore, the image quality improvement engine of the present embodiment is configured to accept an input of a small-sized image and output a large-sized image.

[0384] In this regard, with respect to the paired groups constituting the training data used by the image quality improvement engine, the input data constituting each paired group is a small-sized image, and the output data is a large-sized image. Note that the large-sized image used for the output data can be obtained from a device having higher performance than the camera device that obtains the small-sized image, or can be obtained by changing the settings of the camera device. In addition, in the case where a group of large-sized images already exists, the size of each image in the group of large-sized images can be reduced to the image size of the image assumed to be obtained from the camera device 10, thereby obtaining a group of small-sized images to be used as input data. In addition, with respect to the large-sized images, images obtained by superimposing small-sized images in a manner similar to the first embodiment and the like can be used.

[0385] Note that regarding enlarging the image size of the input image by the image quality improving unit 404 according to the present embodiment, since the image as training data is obtained from a device having higher performance than the image pickup device 10 or is obtained by changing the settings of the image pickup device 10, the enlargement in this case is different from the simple enlargement of the image. Specifically, the process for enlarging the image size of the input image by the image quality improving unit 404 according to the present embodiment can reduce the degradation of the resolution compared with the case where the image is simply enlarged.

[0386] According to this configuration, the image quality improving unit 404 according to the present embodiment can generate a high-quality image generated by reducing noise and enhancing the contrast of an input image and also increasing the image size.

[0387] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that since the processing in step S520, step S530, and step S550 according to the present embodiment is the same as the processing in these steps in the first embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0388] In step S510, the obtaining unit 401 obtains a small-sized image as a processing target from the camera 10 or other devices as input data. Note that the obtaining unit 401 can obtain a small-sized image generated by the camera 10 as an input image in response to a request from the camera 10. For example, such a request can be issued when the camera 10 generates an image, or when the image generated by the camera 10 is displayed on the display unit 20 before being stored in a storage device included in the camera 10, or when the stored image is displayed on the display unit 20 after being stored in the storage device, or when a high-quality image is used for image analysis processing.

[0389] Since the processing in step S520 and step S530 is the same as that in the first embodiment, the description of the processing is omitted here. In step S540, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, thereby generating an image that has undergone image size increase, noise reduction, and contrast enhancement as a high-quality image. The processing thereafter is the same as that in the first embodiment, so the description of the processing is omitted here.

[0390] As described above, the image quality improvement unit 404 according to the present embodiment generates a high-quality image, which is enlarged in image size and subjected to at least one of noise reduction and contrast enhancement compared to the input image. In this way, the image processing device 400 according to the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness to the subject or the labor of the person performing the imaging.

[0391] Note that although a high-quality image is generated in this embodiment by performing the image quality improvement processing according to the first embodiment, etc. and the processing for improving the resolution by a single image quality improvement engine, the configuration for performing such processing is not limited to this configuration. For example, the image quality improvement unit may include an image quality improvement engine that performs the image quality improvement processing according to the first embodiment, etc. and a separate image quality improvement engine that performs the image size increase processing.

[0392] In this case, the image quality improvement engine that performs the image quality improvement processing according to the first embodiment, etc. can use a machine learning model that performs learning similarly to the image quality improvement engine according to the first embodiment, etc. In addition, the high-quality image generated by the image quality improvement engine according to the first embodiment, etc. is used as input data of training data for the image quality improvement engine that performs the image size increase processing. In addition, the high-quality image generated by the image quality improvement engine according to the first embodiment, etc. for the image obtained by the high-performance camera device is used as input data of the training data of the relevant image quality improvement engine. In this way, the image quality improvement engine that performs the image size increase processing can generate a final high-quality image, which is an image generated by increasing the image size of the high-quality image generated by performing the image quality improvement processing according to the first embodiment, etc.

[0393] Furthermore, image size increasing processing by the relevant image quality improvement engine may be performed before image quality improvement processing by the image quality improvement engine according to the first embodiment and the like. In this case, training data for the image quality improvement engine for performing the image size increasing processing is composed of a pairing group composed of input data as a small-size image obtained by the camera device and output data as a large-size image. Furthermore, training data for the image quality improvement engine for performing the image quality improvement processing according to the first embodiment and the like is composed of a pairing group composed of input data as a large-size image and output data as an image generated by superimposing the large-size images.

[0394] Also with this configuration, the image processing device 400 can generate an image whose image size is enlarged compared to the input image and which is subjected to at least one of noise reduction and contrast enhancement as a high-quality image.

[0395] Note that although the present embodiment describes a configuration in which the output data of the superimposed image is used as training data for the image quality improvement processing according to the first embodiment and the like, the output data is not limited thereto, similar to the first embodiment. In other words, a high-quality image obtained by performing at least one of the imaging methods and processing groups including the following processing can be used: superimposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0396] Note that, similar to the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or other devices connected to the image processing device 400 .

[0397] <Sixteenth Embodiment>

[0398] Next, refer to Figure 4 and Figure 5 An image processing apparatus according to a sixteenth embodiment is described. In this embodiment, an image quality improvement unit performs spatial resolution improvement as well as image quality improvement according to the first embodiment and the like.

[0399] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0400] The acquisition unit 401 according to this embodiment acquires a low spatial resolution image as an input image. Note that the term “low spatial resolution image” refers to an image with a spatial resolution lower than that of the high spatial resolution image output by the image quality improvement unit 404 .

[0401] Similar to the first embodiment, the image quality improvement unit 404 includes an image quality improvement engine. However, the image quality improvement engine of this embodiment is configured to improve the spatial resolution of the input image while reducing noise and enhancing the contrast of the input image. Therefore, the image quality improvement engine of this embodiment is configured to accept the input of a low spatial resolution image and output a high spatial resolution image.

[0402] In this regard, with respect to the pairing groups constituting the training data used by the image quality improvement engine, the input data constituting each pairing is a low spatial resolution image, and the output data is a high spatial resolution image. Note that the high spatial resolution image can be obtained from a device having higher performance than the camera device that obtains the low spatial resolution image, or can be obtained by changing the settings of the camera device. In addition, with respect to the high spatial resolution image, an image obtained by superimposing a low spatial resolution image in a manner similar to the first embodiment and the like can be used.

[0403] According to this configuration, the image quality improving unit 404 according to the present embodiment can generate a high-quality image generated by reducing noise and enhancing the contrast of an input image and also improving spatial resolution.

[0404] Next, refer to Figure 5A series of image processing operations according to the present embodiment are described. Note that since the processing in step S520, step S530, and step S550 according to the present embodiment is the same as the processing in these steps in the first embodiment, the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S520 is performed, the processing in step S530 may be omitted, and the processing may proceed to step S540.

[0405] In step S510, the obtaining unit 401 obtains a low spatial resolution image as a processing target from the camera 10 or other devices as input data. Note that the obtaining unit 401 can obtain a low spatial resolution image generated by the camera 10 as an input image in response to a request from the camera 10. For example, such a request can be issued when the camera 10 generates an image, or when the image generated by the camera 10 is displayed on the display unit 20 before being stored in a storage device included in the camera 10, or when the stored image is displayed on the display unit 20 after being stored in the storage device, or when a high-quality image is used for image analysis processing.

[0406] Since the processing in step S520 and step S530 is the same as that in the first embodiment, the description of the processing is omitted here. In step S540, the image quality improvement unit 404 inputs the input image to the image quality improvement engine, thereby generating an image with improved spatial resolution, noise reduction and contrast enhancement as a high-quality image. The processing thereafter is the same as that in the first embodiment, so the description of the processing is omitted here.

[0407] As described above, the image quality improvement unit 404 according to the present embodiment generates an image as a high-quality image, which has improved spatial resolution compared to the input image and has undergone at least one of noise reduction and contrast enhancement. In this way, the image processing device 400 according to the present embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness to the subject or the labor of the person performing the imaging.

[0408] Note that although a high-quality image is generated in the present embodiment by performing the image quality improvement processing according to the first embodiment and the like and the processing for improving the resolution by a single image quality improvement engine, the configuration for performing such processing is not limited to the aforementioned configuration. For example, the image quality improvement unit may include an image quality improvement engine that performs the image quality improvement processing according to the first embodiment and the like and a separate image quality improvement engine that performs the resolution improvement processing.

[0409] In this case, the image quality improvement engine that performs the image quality improvement processing according to the first embodiment, etc. can use a machine learning model that performs learning similarly to the image quality improvement engine according to the first embodiment, etc. In addition, the high-quality image generated by the image quality improvement engine according to the first embodiment, etc. is used as input data of training data for the image quality improvement engine that performs resolution improvement processing. In addition, the high-quality image generated by the image quality improvement engine according to the first embodiment, etc. for the image obtained by the high-performance camera device is used as input data of the training data of the relevant image quality improvement engine. In this way, the image quality improvement engine that performs the spatial resolution improvement processing can generate a final high-quality image, which is an image generated by improving the spatial resolution of the high-quality image generated by performing the image quality improvement processing according to the first embodiment, etc.

[0410] Furthermore, spatial resolution improvement processing by the relevant image quality improvement engine may be performed before image quality improvement processing by the image quality improvement engine according to the first embodiment, etc. In this case, training data for the image quality improvement engine that performs spatial resolution improvement processing is composed of a pairing group composed of input data as a low spatial resolution image obtained by a camera device and output data as a high spatial resolution image. Furthermore, training data for the image quality improvement engine that performs image quality improvement processing according to the first embodiment, etc. is composed of a pairing group composed of input data as a high spatial resolution image and output data as an image generated by superimposing the high spatial resolution images.

[0411] Also with this configuration, the image processing device 400 can generate an image as a high-quality image in which the spatial resolution is improved compared to the input image and at least one of noise reduction and contrast enhancement is performed.

[0412] Note that, although the configuration of output data using a superimposed image as training data in the image quality improvement processing according to the first embodiment and the like is described in this embodiment, the output data is not limited thereto, similar to the first embodiment. In other words, a high-quality image obtained by performing at least one of the imaging methods and processing groups including the following processing can be used: superimposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, imaging using a high-performance imaging device, high-cost processing, and noise reduction processing.

[0413] In addition, in addition to the spatial resolution improvement process, the image quality improvement unit 404 can also use the image quality improvement engine to perform the image quality improvement process according to the fifteenth embodiment. In this case, the following image can be generated as a high-quality image, which has undergone an increase in image size and an increase in spatial resolution compared to the input image, and has also undergone at least one of noise reduction and contrast enhancement. In this way, the image processing device 400 according to this embodiment can output a high-quality image suitable for image diagnosis without increasing the invasiveness to the subject or the labor of the person performing the imaging.

[0414] Note that, similar to the first embodiment, the output unit 405 may output the generated high-quality image to the imaging device 10 or other devices connected to the image processing device 400 .

[0415] <Seventeenth Embodiment>

[0416] Next, refer to Fig. 22 and Fig.23 An image processing apparatus according to a seventeenth embodiment is described. In this embodiment, the analyzing unit performs image analysis on the high-quality image generated by the image quality improving unit.

[0417] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to this embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to this embodiment is described below centering on the differences from the image processing apparatus according to the first embodiment.

[0418] Fig. 22 2200 according to the present embodiment. In addition to the acquisition unit 401, the imaging condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405, the image processing device 2200 according to the present embodiment is also equipped with an analysis unit 2208. Note that the image processing device 2200 may be composed of a plurality of devices, each of which is equipped with one or more of these components. Here, since the configurations of the acquisition unit 401, the imaging condition acquisition unit 402, the image quality improvement possibility determination unit 403, the image quality improvement unit 404, and the output unit 405 are the same as those of the image processing device according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted below.

[0419] The analysis unit 2208 performs predetermined image analysis processing on the high-quality image generated by the image quality improvement unit 404. For example, in the field of ophthalmology, for an image obtained by OCT, examples of image analysis processing include any existing image analysis processing, such as segmentation of retinal layers, layer thickness measurement, three-dimensional shape analysis of the papilla, lamina cribrosa analysis, blood vessel density measurement of OCTA images, and corneal shape analysis. In addition, the image analysis processing is not limited to the analysis processing in the field of ophthalmology, and, for example, also includes any existing analysis processing in the field of radiation, such as diffusion tensor analysis or VBL (voxel-based morphometry) analysis.

[0420] The output unit 405 may cause the analysis result obtained by the image analysis processing of the analysis unit 2208 to be displayed on the display unit 20 together with the display of the high-quality image generated by the image quality improvement unit 404. Note that the output unit 405 may cause the display unit 20 to display only the image analysis result obtained by the analysis unit 2208, or may output the image analysis result to the camera 10 or an image management system or other device, etc. Note that the form of displaying the analysis result may be any form corresponding to the image analysis processing performed by the analysis unit 2208, and for example, the analysis result may be displayed as an image, a numerical value, or a character. In addition, the form of displaying the analysis result may be a form in which the analysis result obtained by performing analysis processing on the high-quality image is displayed on the high-quality image in an overlaid manner with any transparency. In other words, the analysis result may be displayed in the form of an image (e.g., a two-dimensional graph) obtained by performing fusion processing on the high-quality image and the analysis result obtained by performing analysis processing on the high-quality image with any transparency.

[0421] The following is an example of OCTA in-plane image. Fig.23 A series of image processing operations according to the present embodiment is described. Fig.23 is a flowchart showing a series of image processing operations according to the present embodiment. Note that the processing in steps S2310 to S2340 according to the present embodiment is the same as the processing in steps S510 to S540 in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S2320 is performed, the processing in step S2330 may be omitted, and the processing may proceed to step S2340.

[0422] In step S2340, the image quality improvement unit 404 performs image quality improvement on the OCTA in-plane image, and then the process proceeds to step S2350. In step S2350, the analysis unit 2208 performs image analysis on the high-quality image generated in step S2340. As an image analysis of the OCTA in-plane image that has been subjected to image quality improvement, a place corresponding to a blood vessel (vascular area) can be detected from the image by applying any binarization process. The area density can be analyzed by determining the ratio of the image occupied by the detected place corresponding to the blood vessel. In addition, by thinning the place corresponding to the binarized blood vessel, an image with a line width of 1 pixel can be obtained, and the ratio occupied by the blood vessel not depending on the thickness (also called "skeleton density") can be determined. A configuration can be adopted so that the area and shape (circularity, etc.) of the avascular zone (FAZ) can be analyzed using these images. As an analysis method, the above-mentioned numerical values ​​can be calculated from the entire image, or the numerical values ​​can be calculated for a designated region of interest (ROI) based on the instruction of the inspector (user) input using a user interface (not shown). The ROI setting does not necessarily have to be specified only by the examiner, and the ROI may be a predetermined area that is automatically specified. The various parameters described above are examples of analysis results related to blood vessels, and any parameter may be used as long as it is a parameter related to blood vessels. Note that the analysis unit 2208 can perform a variety of image analysis processes. In other words, although an analysis example related to an OCTA in-plane image is described here, the type of analysis that can be performed is not limited to this, and analyses such as retinal layer segmentation, layer thickness measurement, three-dimensional shape analysis of the papilla, and lamina cribrosa analysis may be performed simultaneously on images obtained by OCT. In this regard, the analysis unit 2208 may perform some or all of the various image analysis processes in response to instructions input by the examiner through any input device.

[0423] In step S2360, the output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improvement unit 404 and the analysis result obtained by the analysis unit 2208. Note that the output unit 405 may output the high-quality image and the analysis result to separate display units or devices. In addition, the output unit 405 may cause the display unit 20 to display only the analysis result. In addition, in the case where the analysis unit 2208 outputs a plurality of analysis results, the output unit 405 may output some or all of the plurality of analysis results to the display unit 20 or other devices. For example, the output unit 405 may cause the display unit 20 to display the analysis result related to the blood vessels in the OCTA in-plane image as a two-dimensional graph. In addition, the output unit 405 may superimpose the value representing the analysis result related to the blood vessels in the OCTA in-plane image on the OCTA in-plane image, and cause the display unit 20 to display the resulting image.

[0424] As described above, the image processing apparatus 2200 according to the present embodiment further includes the analysis unit 2208 that performs image analysis on the high-quality image, and the output unit 405 causes the analysis result obtained by the analysis unit 2208 to be displayed on the display unit 20. Therefore, since the high-quality image is used for the image analysis performed by the image processing apparatus 2200 according to the present embodiment, the accuracy of the analysis can be improved.

[0425] In addition, similar to the first embodiment, the output unit 405 can output the generated high-quality image to the camera 10 or other devices connected to the image processing device 2200. In addition, similar to the first embodiment, the output data of the training data of the image quality improvement engine is not limited to the high-quality image obtained by performing the superposition process. In other words, a high-quality image obtained by performing at least one of the camera methods and processing groups including the following processes can be used: superposition processing, MAP estimation processing, smoothing filter processing, grayscale conversion processing, camera using a high-performance camera, high-cost processing, and noise reduction processing.

[0426] <Eighteenth Embodiment>

[0427] Next, refer to Figure 4 An image processing apparatus according to an eighteenth embodiment is described. In this embodiment, an example is described in which an image quality improvement unit generates a high-quality image by adding noise to an image when performing learning and learning the noise component.

[0428] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0429] The acquisition unit 401 according to this embodiment acquires an image to be processed as input data from the camera 10 or other devices. Fig.24 A configuration example of a CNN in the image quality improvement unit according to the present embodiment is described. Fig.24 is a diagram showing an example of the construction of a machine learning model in the image quality improvement unit 404. Fig.24 The construction shown in consists of multiple layer groups that are responsible for processing the input value groups for output. Note that Fig.24As shown in , the types of layers included in the construction are convolution layers, downsampling layers, upsampling layers, and merging layers. The convolution layer is a layer that performs convolution processing on the input value group according to the set parameters such as the kernel size of the filter, the number of filters, the value of the stride, and the dilation value. Please note that the dimension of the kernel size of the filter can be changed according to the dimension of the input image. The downsampling layer performs processing for making the number of output value groups less than the number of input value groups by sparsely pruning or combining input value groups. Specifically, for example, the processing is a maximum pooling processing. The upsampling layer performs processing for making the number of output value groups greater than the number of input value groups by copying the input value group or adding values ​​interpolated from the input value group. Specifically, for example, the processing is a linear interpolation processing. The merging layer is a layer that performs the following processing: value groups such as the output value group of a certain layer or the pixel value group constituting the image are input from multiple sources, and these value groups are merged by cascading or adding value groups. According to this configuration, the value group output when the pixel value group constituting the input image Im2410 is processed by the convolution processing block and the pixel value group constituting the input image Im2410 are merged by the merging layer. Thereafter, the merged pixel value group is formed into a high-quality image Im2420 by the final convolution layer. Note that although not shown in the drawings, as a modified example of the configuration of the CNN, for example, a batch normalization layer or an activation layer using a rectifier linear unit may be incorporated after the convolution layer or the like.

[0430] The image quality improvement engine of the present embodiment is trained by receiving an input of a low-quality image generated by adding a first noise component to an image obtained from the camera 10 or other devices, and adopts an image generated as a high-quality image by adding a second noise component to the image obtained from the camera 10 or other devices as output data. In other words, as training images during learning according to the present embodiment, ordinary images are used as low-quality images and high-quality images, and the respective noise components of the respective images are different from each other. Because the same image is used as a low-quality image and a high-quality image, alignment is not required when forming an image pair.

[0431] Regarding the noise component, Gaussian noise or noise modeled with characteristic noise of the target image, etc. is added as noise. However, the first noise and the second noise are noises different from each other. The term "different noise" means that the spatial position (pixel position) of the added noise is different, or the value of the noise is different, etc. As the characteristic noise of the target image, for example, in the case of OCT, the noise can be estimated based on the data obtained by imaging in the absence of a model eye or an eye to be examined, and it can be used as a noise model. In the case of OCTA, noise based on noise appearing within the range of the avascular zone (FAZ), or noise appearing in an image obtained by imaging a model eye that schematically reproduces blood flow, can be used as a noise model.

[0432] In the case of Gaussian noise, the standard deviation or variance value is defined as the size of the noise, and the noise is randomly applied to the image based on these values. A structure can be adopted so that the average value does not change as a whole due to the application of random noise. In other words, a structure is adopted so that the average value of the noise added to each pixel of an image is zero. In this case, it is not necessary to adopt a structure that makes the average value become 0, and it is sufficient to add noises with different patterns to the input data and the output data. In addition, it is not necessary to add noise to both the input data and the output data, and noise can be added to one of the input data and the output data. In the case of not adding noise, for example, although a false image of a blood vessel may appear in the image after the image quality is improved, it can also be considered that this occurs when the difference between the images before and after the image quality is improved is relatively large. Therefore, a structure can be adopted that reduces the difference between the images before and after the image quality is improved. At this time, when learning is performed, two images obtained by adding noises with different patterns to a low-quality image and a high-quality image can be used as an image pair, or two images obtained by adding noises with different patterns to a high-quality image can be used as an image pair.

[0433] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404. Note that the output unit 405 may cause the display unit 20 to display the input image together with the high-quality image.

[0434] The processing thereafter is the same as that in the first embodiment, so the description of the processing is omitted here.

[0435] Note that, although in the present embodiment, a high-quality image is generated using an image generated by obtaining a low-quality image from the camera 10 or other device and adding a first noise component and a second noise component different from the first noise to the low-quality image, respectively, the configuration for performing these processing operations is not limited to the aforementioned configuration. For example, for an image to which noise is added, a configuration may be adopted in which the first noise component and the second noise component are added to a high-quality image to which the superposition processing as described in the first embodiment is performed. In other words, a configuration may be adopted in which learning is performed in which an image obtained by adding the first noise component to an image generated by the superposition processing is adopted as a low-quality image, and an image obtained by adding the second noise component to the image generated by the superposition processing is adopted as a high-quality image.

[0436] In addition, although an example of learning using the first noise component and the second noise component is described in the present embodiment, the learning method is not limited thereto. For example, a configuration may be adopted in which learning is performed in such a manner that only the first noise component is added to an image to be used as a low-quality image and no noise component is added to an image to be used as a high-quality image. The image used at this time may be an image obtained from the camera 10 or other devices, or an image generated by performing superposition processing on the above-mentioned image as a processing object.

[0437] In addition, for various input images or for each rectangular area image to be learned, the size of the noise component can be changed dynamically. Specifically, increasing noise with a large value increases the effect of removing noise, and increasing noise with a small value reduces the effect of removing noise. Therefore, for example, a structure can be adopted in which learning is performed in the following manner: the added noise is adjusted according to the condition or type of the entire image or rectangular area image, for example, the value of the noise component to be added is reduced in the case of a dark image and the value of the noise component to be added is increased in the case of a bright image.

[0438] Note that although imaging conditions of images are not explicitly specified in this embodiment, learning is performed using images of various imaging ranges obtained by different numbers of scans, frontal images of different imaging sites and different depths, and the like.

[0439] In the above description, an image obtained from the camera 10 or other devices, a noise image generated by adding noise to the aforementioned image, an image generated by superposition processing, and an image generated by adding noise to the image generated by superposition processing have been described. However, the combination of these images is not limited to the combination described above, and any combination of low-quality images and high-quality images may be used.

[0440] <Nineteenth Embodiment>

[0441] Next, refer to Fig.25 and Fig.26 An image processing apparatus according to a nineteenth embodiment is described. In this embodiment, an image quality improvement unit includes a plurality of image quality improvement engines, and generates a plurality of high-quality images for an input image. In addition, an example is described in which a combination unit 2505 combines a plurality of high-quality images output from a plurality of image quality improvement engines.

[0442] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0443] The obtaining unit 401 according to the present embodiment obtains an image as a processing target from the imaging device 10 or other devices as input data.

[0444] Similar to the second embodiment, the image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines. Each of the plurality of image quality improvement engines uses learning data different from each other regarding at least one of the imaging conditions of the imaging part, the imaging angle of view, the front image at different depths, the noise component, and the image resolution for learning. The image quality improvement unit 404 generates a high-quality image using the plurality of image quality improvement engines according to at least one of the imaging conditions of the imaging part, the imaging angle of view, the front image at different depths, the noise component, and the image resolution of the input image.

[0445] Fig.26 is a flowchart showing a series of image processing operations according to the present embodiment. Note that the processing in step S2610 and step S2620 according to the present embodiment is the same as the processing in step S510 and step S520 in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S2620 is performed, the processing in step S2630 may be omitted, and the processing may proceed to step S2640.

[0446] In step S2620, similarly to the first embodiment, when the imaging condition obtaining unit 402 obtains the imaging condition group of the input image, the process proceeds to step S2630. In step S2630, similarly to the second embodiment, the image quality improvement possibility determination unit 403 uses the obtained imaging condition group to determine whether any image quality improvement engine included in the image quality improvement unit 404 can process the input image.

[0447] If the image quality improvement possibility determination unit 403 determines that none of the image quality improvement engine group can process the input image, the process proceeds to step S2660. On the other hand, if the image quality improvement possibility determination unit 403 determines that any one of the image quality improvement engine group can process the input image, the process proceeds to step S2640. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, similar to the first embodiment, even if it is determined that some imaging conditions cannot be processed by any image quality improvement engine, the process in step S2640 may be performed.

[0448] In step S2640, the image quality improving unit 404 inputs the input image obtained in step S2610 to each image quality improving engine in the image quality improving engine group, thereby generating a high-quality image group.

[0449] In step S2650, the combining unit 2405 combines several high-quality images in the high-quality image group generated in step S2640. Specifically, for example, the results of two high-quality images are combined, which are a high-quality image generated as a result of processing by a first image quality improvement engine that uses an image pair consisting of a low-quality image obtained from the camera 10 and a high-quality image obtained by performing a superposition process such as additive averaging on an image group obtained by photographing the low-quality image multiple times as described in the first embodiment for learning, and a high-quality image generated as a result of processing by a second image quality improvement engine that uses an image pair obtained by adding noise to an image as described in the eighteenth embodiment for learning. As a combining method, a method of combining using additive averaging or weighted additive averaging or the like can be adopted.

[0450] In step S2660, the output unit 405 causes the image generated by combining the images in step S2650 to be displayed on the display unit 20, or outputs the image to other devices. However, if it is determined in step S2630 that it is impossible to process the input image, the output unit 405 outputs the input image as the output image. Note that in the case where the examiner instructs to display the input image or in the case where it is impossible to process the input image, the output unit 405 can cause the display unit 20 to display information indicating that the output image is the same as the input image.

[0451] <Twentieth Embodiment>

[0452] Next, refer to Figure 4 An image processing apparatus according to a twentieth embodiment is described. In this embodiment, an example of an image quality improving unit is described in which a second image quality improving engine generates a high-quality image using an output result of a first image quality improving engine.

[0453] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0454] The obtaining unit 401 according to the present embodiment obtains an image as a processing target from the imaging device 10 or other devices as input data.

[0455] The image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines that are the same as the image quality improvement engines provided in the first embodiment. The image quality improvement unit of the present embodiment includes a first image quality improvement engine that uses a low-quality image obtained from the camera device 10 or other devices as input data and a medium-quality image generated from a plurality of low-quality images as output data for learning. In addition, the image quality improvement unit includes a second image quality improvement engine that uses an image output from the first image quality improvement engine and an image having an image quality higher than that of the medium-quality image as output data for learning. Note that since the medium-quality image is the same as that in the fourteenth embodiment, the description of the medium-quality image is omitted here.

[0456] The output unit 405 causes the display unit 20 to display the high-quality image generated by the image quality improving unit 404. Note that the output unit 405 may cause the display unit 20 to display the input image together with the high-quality image, and in this case, the output unit 405 may cause the display unit 20 to display information indicating that the input image is an image generated from a plurality of low-quality images.

[0457] Next, refer to Figure 5 A series of image processing operations according to the present embodiment are described. Note that the processing in steps S510 to S530 according to the present embodiment is the same as that in the first embodiment, so the description of the processing is omitted here.

[0458] In step S540, the image quality improvement unit 404 uses the image quality improvement engine to improve the image quality of the input image, thereby generating a high-quality image that is more suitable for image diagnosis than the input image. Specifically, the image quality improvement unit 404 inputs the input image to the first image quality improvement engine so that the first image quality improvement engine generates a first high-quality image with improved image quality. In addition, the image quality improvement unit 404 inputs the first high-quality image to the second image quality improvement engine to obtain a second high-quality image. Each image quality improvement engine generates a high-quality image obtained by performing superposition processing using the input image based on a machine learning model that performs machine learning using training data. Therefore, compared with the input image, each image quality improvement engine can generate a high-quality image with reduced noise or enhanced contrast.

[0459] The processing thereafter is the same as that in the first embodiment, so the description of the processing is omitted here.

[0460] Note that although in this embodiment, a high-quality image is generated using a first image quality improvement engine that learns using a pair consisting of a low-quality image and a medium-quality image obtained from the camera device 10 or other devices, and a second image quality engine that learns using a pair consisting of a first high-quality image and a high-quality image, the configuration for performing such processing is not limited to the aforementioned configuration. For example, the following configuration may be adopted: a pair of images used for learning by the first image quality improvement engine is an image used by the engine described in the eighteenth embodiment to learn noise, and the second image quality improvement engine learns using a pair consisting of a first high-quality image and a high-quality image. As an opposite configuration, a configuration including a first image quality improvement engine that learns using a pair consisting of a low-quality image and a medium-quality image, and a second image quality improvement engine that learns using an image obtained by adding noise to the first high-quality image may be adopted.

[0461] In addition, the first image quality improvement engine and the second image quality improvement engine may each be constructed as an engine that uses noise learning as described in the eighteenth embodiment. In this case, for example, the first image quality improvement engine uses a pair consisting of an image obtained by adding the first noise and the second noise to a high-quality image for learning, the high-quality image being generated using an image generated by superposition processing, and the second image quality improvement engine uses a pair consisting of an image obtained by adding the first noise and the second noise to the first high-quality image generated by the first image quality improvement engine for learning. Note that although a construction including two image quality improvement engines is described in this embodiment, the present invention is not limited thereto, and a construction in which a third engine and a fourth engine are also connected for processing may be adopted. By cleaning the image to be used for learning, a network that can easily generate smoother and clearer images is constructed.

[0462] <Twenty-first embodiment>

[0463] Next, refer to Figure 4 and Fig. 27 An image processing apparatus according to the twenty-first embodiment is described. In the first embodiment, the image quality improvement unit 404 includes one image quality improvement engine. In contrast, in the present embodiment, the image quality improvement unit includes a plurality of image quality improvement engines that perform machine learning using training data different from each other, and generates a plurality of high-quality images for an input image.

[0464] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the second embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatuses according to the first and second embodiments. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatuses according to the first and second embodiments, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment and the second embodiment, and description of these components is omitted hereinafter.

[0465] The image quality improvement unit 404 according to the present embodiment includes two or more image quality improvement engines that perform machine learning using training data that are different from each other. A method for creating a training data group according to the present embodiment will now be described. First, a pairing group consisting of a source image as input data and a superimposed image as output data is prepared by performing imaging to obtain images having various imaging ranges and having different scan times. Taking OCT or OCTA as an example, for example, a pairing of a first image group obtained by imaging 300 A scans and 300 B scans within a range of 3×3 mm and a pairing of a second image group obtained by imaging 500 A scans and 500 B scans within a range of 10×10 mm are created. At this time, there is a difference of two times in scanning density between the pairing of the first image group and the pairing of the second image group. Therefore, these image groups are grouped into separate groups. In addition, in the case where there is an image group obtained by imaging 600 A scans and 600 B scans within a range of 6×6 mm, the image group is grouped into the same group as the first image group. In other words, in this case, image groups having the same or almost the same (with a tolerance of about 10%) scan densities are grouped in the same group.

[0466] Next, the paired groups are grouped for each scanning density to create a training data set. For example, a training data set is created by creating first training data consisting of paired groups obtained by imaging at a first scanning density, and creating second training data consisting of paired groups obtained by imaging at a second scanning density.

[0467] Thereafter, each image quality improvement engine is made to perform machine learning using each training data set. For example, an image quality improvement engine group is prepared, the image quality improvement engine group including a first image quality improvement engine corresponding to a machine learning model trained using the first training data and a second image quality improvement engine corresponding to a machine learning model trained using the second training data.

[0468] Because different training data is used to train the respective machine learning models corresponding to each of these image quality improvement engines, the degree to which each of these image quality improvement engines can improve the image quality of the input image input to the image quality improvement engine will differ depending on the imaging conditions of the input image. Specifically, in the case of the first image quality improvement engine, the degree of image quality improvement for the input image obtained by imaging at the first scanning density is high, and the degree of image quality improvement for the image obtained by imaging at the second scanning density is low. Similarly, in the case of the second image quality improvement engine, the degree of image quality improvement for the input image obtained by imaging at the second scanning density is high, and the degree of image quality improvement for the image obtained by imaging at the first scanning density is low.

[0469] On the other hand, in some cases, a sufficient number of images of various imaging ranges and different scanning densities cannot be collected as training data when learning. In this case, as described in the eighteenth embodiment, an image quality improvement engine that has learned the noise component is prepared for these image groups.

[0470] Since the image quality improvement engine that has learned the noise component is not easily affected by the scanning density at the time of image capturing, when an image with a scanning density that has not been learned is input, the image quality improvement engine in question is applied.

[0471] Since each training data set is composed of paired groups grouped according to scanning density, the image quality trends of the image groups constituting the relevant paired groups are similar to each other. Therefore, if the scanning density corresponds to the training data used to train the image quality improvement engine in question, the image quality improvement engine can perform image quality improvement more effectively than the image improvement engine according to the first embodiment. Note that the imaging conditions used to group paired training data are not limited to the scanning density, and may be imaging sites or images at different depths with respect to frontal images, or a combination of two or more of these conditions.

[0472] The following will refer to Fig. 27 A series of image processing operations according to the present embodiment is described. Fig. 27 27 is a flowchart showing a series of image processing operations according to the present embodiment. Note that since the processing in step S2710 and step S2720 is the same as the processing in step S510 and step S520 according to the first embodiment, the description of the processing is omitted here.

[0473] After obtaining the imaging conditions of the input image in step S2720, the process proceeds to step S2730. In step S2730, the image quality improvement possibility determination unit 403 uses the imaging condition group obtained in step S2720 to determine whether any one of the image quality improvement engine groups included in the image quality improvement unit 404 can process the input image.

[0474] If the image quality improvement possibility determination unit 403 determines that the imaging condition is outside the processable imaging condition range, the process proceeds to step S2770. On the other hand, if the image quality improvement possibility determination unit 403 determines that the imaging condition is within the processable imaging condition range, the process proceeds to step S2740.

[0475] In step S2740, the image quality improvement unit 404 selects an image quality improvement engine from the image quality improvement engine group to perform image quality improvement processing based on the imaging condition of the input image obtained in step S2720 and the information related to the training data of the image quality improvement engine group. Specifically, for example, the image quality improvement unit 404 selects an image quality improvement engine that has information of the training data related to the scanning density in the imaging condition group obtained in step S2720 and has a high degree of image quality improvement. In the above example, if the scanning density is the first scanning density, the image quality improvement unit 404 selects the first image quality improvement engine.

[0476] On the other hand, in step S2770, the image quality improving unit 404 selects the image quality improving engine that has learned the noise component.

[0477] In step S2750, the image quality improvement unit 404 generates a high-quality image by performing image quality improvement on the input image using the image quality improvement engine selected in step S2740 or step S2770. Thereafter, in step S2760, the output unit 405 outputs the high-quality image generated in step S2750 and causes the display unit 20 to display the high-quality image. Note that when causing the display unit 20 to display the high-quality image, the output unit 405 may also cause the display unit 20 to display information indicating that the high-quality image is generated using the image quality improvement engine selected by the image quality improvement unit 404.

[0478] As described above, the image quality improvement unit 404 according to the present embodiment includes a plurality of image quality improvement engines that learn using learning data that are different from each other. Here, each of the plurality of image quality improvement engines learns using learning data that are different from each other regarding at least one of the imaging conditions of the imaging part, the imaging angle of view, the frontal image at different depths, and the image resolution. In addition, for data that cannot fully collect the correct answer data (output data), the image quality improvement engine is an engine that has learned using a noise component. The image quality improvement unit 404 generates a high-quality image using an image quality improvement engine corresponding to at least one of these imaging conditions.

[0479] With this configuration, the image processing apparatus 400 according to the present embodiment can generate a high-quality image more effectively.

[0480] <Twenty-second embodiment>

[0481] Next, refer to Figures 30 to 32C An image processing apparatus according to a twenty-second embodiment is described. In this embodiment, a wide-angle image generation unit generates a wide-angle image (panoramic image) using a plurality of high-quality images generated by an image quality improvement unit.

[0482] Unless otherwise explicitly stated, the configuration and processing of the image processing apparatus according to the present embodiment are the same as those of the image processing apparatus 400 according to the first embodiment. Therefore, the image processing apparatus according to the present embodiment will be described below centering on the differences from the image processing apparatus according to the first embodiment. Note that since the configuration of the image processing apparatus according to the present embodiment is the same as that of the image processing apparatus according to the first embodiment, Figure 4 The components shown in FIG. 1 are denoted by the same reference numerals as those in the first embodiment, and description of these components will be omitted hereinafter.

[0483] Fig.31Ais a flowchart showing a series of image processing operations according to the present embodiment. In step S3110, the acquisition unit 401 obtains a plurality of images (at least two images) as input images from the imaging device 10 or other devices. The plurality of images are images obtained by imaging different positions of the same subject (the subject's eye, etc.), and are images obtained by overlapping a portion of the image without imaging a place where the subject completely overlaps. As an example, a case where the subject's eye is imaged will be described. In this case, during imaging, the position of the fixation lamp is changed and the subject's eye is stably stared at the fixation lamp, and as a result, images in which different places of the same subject's eye have been imaged can be obtained. Note that when imaging is performed to obtain an image, it is desirable to perform imaging while changing the position of the fixation lamp so that a ratio of at least about 20% of the overlapping areas of adjacent images is an image of the same place. Fig.32A , an example of an OCTA in-plane image captured by changing the position of a fixation lamp so that a portion of each image overlaps with each other in adjacent images is shown. Fig.32A , an example is shown in which the position of the fixation lamp is changed so that different places are photographed a total of five times. Fig.32A Five images are shown as an example in , however, the number of images is not limited to five, and it is sufficient that the number of images is two or more.

[0484] Note that the processing in step S3120 according to the present embodiment is the same as the processing in step S520 in the first embodiment, so the description of the processing is omitted here. Note that in the case where the image quality of the input image is to be improved unconditionally with respect to the imaging conditions, after the processing in step S3120 is performed, the processing in step S3130 may be omitted, and the processing may proceed to step S3140.

[0485] In step S3120, similarly to the first embodiment, when the imaging condition obtaining unit 402 obtains the imaging condition group of each input image, the process proceeds to step S3130. In step S3130, similarly to the first embodiment, the image quality improvement possibility determination unit 403 uses each obtained imaging condition group to determine whether the image quality improvement engine included in the image quality improvement unit 404 can process each input image.

[0486] If the image quality improvement possibility determination unit 403 determines that the image quality improvement engine cannot process a plurality of input images, the process proceeds to step S3160. On the other hand, if the image quality improvement possibility determination unit 403 determines that the image quality improvement engine can process a plurality of input images, the process proceeds to step S3140. Note that, depending on the arrangement or implementation form of the image processing apparatus 400, similarly to the first embodiment, even if it is determined that some imaging conditions cannot be processed by the image quality improvement engine, the process in step S3140 may be performed.

[0487] In step S3140, the image quality improving unit 404 performs processing on the plurality of input images obtained in step S3110, thereby generating a plurality of high-quality images.

[0488] In step S3150, the wide-angle image generation unit 3005 combines the multiple high-quality images in the high-quality image group generated in step S3140. Specifically, an example taking the OCTA in-plane image as an example will be described. The OCTA in-plane image is photographed so that although the multiple images do not completely overlap, partial areas of adjacent images overlap with each other. Therefore, the wide-angle image generation unit 3005 detects the overlapping area from the multiple OCTA in-plane images and uses the overlapping area for alignment. By modifying the OCTA in-plane images based on the alignment parameters and combining the images, an OCTA in-plane image with a wider range than a single OCTA in-plane image can be generated. At this time, since the image quality of the input multiple OCTA in-plane images was improved in step S3140, the OCTA in-plane image with a wide field of view output in step S3150 has been image-quality improved. Fig.32B An example of an OCTA in-plane image with a wide field of view generated by the wide-angle image generation unit 3005 is shown in FIG. Fig.32B Shown by aligning Fig.32A The five images shown in Figure 1 are examples of how this was generated. Fig.32C Shown in Fig.32A and Fig.32B The correlation between the positions of the images in Fig.32C As shown, image Im3210 is arranged at the center, and images Im3220 to Im3250 are arranged around image Im3210. Note that, regarding the OCTA in-plane image, a plurality of OCTA in-plane images can be generated by setting different depth ranges based on the three-dimensional motion contrast data. Fig.32B An example of a surface layer image with a wide field of view is shown in FIG. 1 , but the present invention is not limited thereto. For example, a configuration may be adopted in which Fig.29AThe OCTA in-plane image (Im2910) of the surface layer shown is aligned, and the parameters obtained from the OCTA in-plane image Im2910 are used to modify the OCTA in-plane images of other depth ranges. Alternatively, a color image is used as an input image for alignment, and a synthetic color image is generated, wherein the RG component of the RGB component is used for the OCTA in-plane image of the surface layer, and the B component is used for the OCTA in-plane image as the alignment object. Then, alignment of the synthetic color OCTA in-plane image obtained by combining multiple layers in the depth range into one image can be performed. Therefore, if only the B component is extracted from the aligned color OCTA in-plane image, an OCTA in-plane image with a wide field of view aligned to the target OCTA in-plane image can be obtained. Note that the object to be improved in image quality is not limited to a two-dimensional OCTA in-plane image, and may be a three-dimensional OCT image or three-dimensional motion contrast data itself. In this case, alignment can be performed with three-dimensional data to generate wide-range three-dimensional data. By cutting out an arbitrary cross section (any plane of XYZ is possible) or an arbitrary depth range (range in the Z direction) from a wide range of three-dimensional data, a high-quality wide-angle image can be generated.

[0489] In step S3160, the output unit 405 causes the image generated by combining the plurality of images in step S3150 to be displayed on the display unit 20, or outputs the image to other devices. However, if it is determined in step S3130 that it is impossible to process the input image, the output unit 405 outputs the input image as the output image. Note that in the case where the examiner instructs to display the input image or in the case where it is impossible to process the input image, the output unit 405 can cause the display unit 20 to display information indicating that the output image is the same as the input image.

[0490] Note that although in the present embodiment, high-quality images are generated from a plurality of input images respectively, and the high-quality images are aligned to generate a final single high-quality wide-angle image, the method for generating a single high-quality image from a plurality of input images is not limited to this method. Fig.31B In the different examples of the image quality improvement processing of the present embodiment shown, a configuration may be adopted such that a wide-angle image is first generated and then image quality improvement processing is performed on the wide-angle image to ultimately generate a single high-quality wide-angle image.

[0491] Now we will use Fig.31B In the following description, the Fig.31A Same processing as described in the section.

[0492] In step S3121, the wide-angle image generation unit 3005 combines the multiple images obtained in step S3110. When the wide-angle image is generated, it is similar to the description of the process in step S3150, although the difference from the process in step S3150 is that the image is in a state before image quality improvement is performed, but the input image is an image obtained from the camera 10 or other devices.

[0493] In step S3151, the image quality improving unit 404 performs processing on the high-quality image generated by the wide-angle image generating unit 3005, thereby generating a single high-quality wide-angle image.

[0494] With this configuration, the image processing device 400 according to the present embodiment can generate a wide-angle high-quality image.

[0495] With regard to the aforementioned first to twenty-second embodiments, basically in response to the image quality improvement unit 404 generating a high-quality image or the analysis unit 2208 outputting the analysis result, the high-quality image is automatically displayed on the display unit 20 by the output unit 405. However, the display of the high-quality image can be performed in response to an instruction from the inspector. For example, the output unit 405 can cause the display unit 20 to display an image selected from the high-quality image generated by the image quality improvement unit 404 and the input image according to an instruction from the inspector. In addition, in response to an instruction from the inspector, the output unit 405 can switch the image displayed on the display unit 20 from the camera image (input image) to a high-quality image. In other words, the output unit 405 can change the display of a low-quality image to the display of a high-quality image in response to an instruction from the inspector. In addition, the output unit 405 can change the display of a high-quality image to the display of a low-quality image in response to an instruction from the inspector. In addition, the image quality improvement unit 404 can start the image quality improvement processing by the image quality improvement engine (input the image into the image quality improvement engine) in response to the instruction from the inspector, and the output unit 405 can cause the display unit 20 to display the high-quality image generated by the image quality improvement unit 404. In contrast, when the camera 10 captures the input image, the image quality improvement engine can automatically generate a high-quality image based on the input image, and the output unit 405 can cause the display unit 20 to display the high-quality image in response to the instruction from the inspector. Note that these processing operations can also be performed similarly for the output of the analysis result. In other words, the output unit 405 can change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to the instruction from the inspector. In addition, the output unit 405 can change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in response to the instruction from the inspector. Naturally, in response to the instruction from the inspector, the output unit 405 can change the display of the analysis result of the low-quality image to the display of the low-quality image. In addition, the output unit 405 may change the display of the low-quality image to the display of the analysis result of the low-quality image in response to an instruction from the inspector. Moreover, the output unit 405 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the inspector. In addition, the output unit 405 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to an ins...

Claims

1. A medical image processing device, comprising: an obtaining unit configured to obtain a first image which is a motion contrast en-face image corresponding to at least a partial depth range of a depth range of the eye to be inspected, the at least partial depth range being determined according to an instruction from an examiner; as well as A generating unit configured to generate a second image having at least one of lower noise and higher contrast compared to the first image using the first image as input data input into an image quality improving engine, wherein the image quality improving engine includes a machine learning engine that has been obtained by using training data, the training data including a third image and a fourth image, the third image being a plurality of motion contrast en-face images corresponding to a plurality of partial depth ranges in a depth range of an eye to be inspected, the fourth image having at least one of lower noise and higher contrast compared to the third image, wherein the plurality of partial depth ranges are different from each other, and The medical image processing apparatus is configured to be capable of determining a depth range different from the plurality of partial depth ranges different from each other as the at least partial depth range.

2. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data including an image obtained by averaging a plurality of motion contrast data of the eye to be inspected.

3. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data, wherein the training data includes images obtained by OCTA imaging by an OCT imaging device having higher performance than the OCT imaging device used to perform OCTA imaging of the third image, or images obtained by an OCTA imaging step including a greater number of steps than the OCTA imaging step used to obtain the third image.

4. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The generating unit generates a second image having at least one of lower noise and higher contrast compared to the first image by dividing the first image into a plurality of two-dimensional images and inputting the plurality of two-dimensional images into an image quality improvement engine and integrating a plurality of output images from the image quality improvement engine.

5. The medical image processing apparatus according to claim 4, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data, the training data including pairs of images having a corresponding positional relationship with each other, and The generation unit divides the first image into a plurality of two-dimensional images having an image size corresponding to an image size of the image pair, and inputs the plurality of two-dimensional images to the image quality improvement engine.

6. The medical image processing apparatus according to claim 4, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data, the training data including an image of a plurality of partial regions for an area including the image and a periphery of the image, the plurality of partial regions being arranged so that parts of adjacent partial regions overlap each other.

7. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data including images obtained by adding noise to the images.

8. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data including image pairs obtained by adding noises having patterns different from each other to the images.

9. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data including an image pair obtained by adding noise having patterns different from each other to an image obtained by averaging a plurality of motion contrast data of a subject's eye.

10. The medical image processing apparatus according to claim 1, further comprising: A wide-angle image generating unit is configured to generate a wide-angle image using a plurality of second images obtained from a plurality of first images obtained by performing OCTA imaging of different positions of an eye to be inspected so that partial areas of adjacent motion contrast en-face images overlap with each other.

11. The medical image processing apparatus according to claim 1, wherein: The image quality improvement engine includes a machine learning engine that has been obtained by using training data, the training data including a plurality of motion contrast en-face images corresponding to different depth ranges of the eye to be inspected; and The obtaining unit obtains a motion contrast en-face image corresponding to a partial depth range including a long depth range of different depth ranges as a first image input to the image quality improvement engine.

12. The medical image processing apparatus according to claim 1, further comprising: a determining unit configured to determine whether the generating unit can generate the second image with respect to the first image using the image quality improvement engine, The determination unit makes the determination based on at least one of a photographing part, a photographing system, a photographing angle of view, and an image size of the first image.

13. The medical image processing apparatus according to claim 1, wherein: The generating unit includes a plurality of image quality improving engines which have been obtained by using training data different from each other, and Each of the plurality of image quality improvement engines has performed learning using mutually different learning data for at least one of an imaging part, an imaging angle of view, and an image resolution; and The generation unit generates a second image having at least one of lower noise and higher contrast than the first image using an image quality improvement engine corresponding to at least one of an imaging site, an imaging angle of view, and an image resolution of the first image.

14. The medical image processing apparatus according to claim 1, wherein: The generating unit includes a plurality of image quality improving engines which have been obtained by using training data different from each other, and The generation unit generates a second image having at least one of lower noise and higher contrast than the first image using the first image as input data to an image quality improvement engine selected according to an inspector's instruction among the plurality of image quality improvement engines.

15. The medical image processing apparatus according to claim 1, wherein: The generating unit includes a plurality of image quality improving engines which have been obtained by using training data different from each other, and a generating unit generating a plurality of second images having at least one of lower noise and higher contrast than the first image using the first image as input data to the plurality of image quality improvement engines; and The medical image processing apparatus outputs at least one image among the plurality of second images according to an instruction of an examiner.

16. The medical image processing apparatus according to claim 1, further comprising: an evaluation unit configured to evaluate image quality of the second image using the second image as input data to a machine learning engine that has been obtained by using training data, the training data including evaluation values ​​obtained by a predetermined evaluation technique, wherein: The generating unit includes a plurality of image quality improving engines which have been obtained by using training data different from each other, The generating unit generates a plurality of second images having at least one of lower noise and higher contrast than the first image using the first image as input data to the plurality of image quality improvement engines, and The medical image processing apparatus outputs at least one image among the plurality of second images according to an evaluation result obtained by the evaluation unit.

17. The medical image processing apparatus according to claim 1, further comprising: an analyzing unit configured to perform image analysis on the second image, and A display control unit is configured to cause an analysis result related to the blood vessel in the second image and information indicating that the second image is an image generated by the image quality improvement engine to be displayed on the display unit.

18. The medical image processing apparatus according to claim 1, wherein: The acquisition unit obtains three-dimensional motion contrast data of the subject's eye from the OCT camera device, and generates a motion contrast en-face image using data of at least a partial range in the depth direction of the subject's eye based on the three-dimensional motion contrast data to obtain a first image input to the image quality improvement engine.

19. The medical image processing apparatus according to claim 1, further comprising a display control unit configured to control the display unit to change the display from one of the display of the first image and the display of the second image to the other according to an instruction from an operator.

20. A system comprising: Optical coherence tomography device; as well as The medical image processing apparatus according to claim 1, which is communicably connected to the optical coherence tomography apparatus, The obtaining unit obtains the first image obtained by using the optical coherence tomography device.

21. A medical image processing method, comprising: obtaining a first image that is a motion contrast en-face image corresponding to at least a portion of a depth range of the eye to be inspected, the at least a portion of the depth range being determined according to an instruction from an examiner; as well as generating a second image having at least one of lower noise and higher contrast compared to the first image using the first image as input data into an image quality improvement engine, wherein the image quality improvement engine includes a machine learning engine that has been obtained by using training data, the training data including a third image and a fourth image, the third image being a plurality of motion contrast en-face images corresponding to a plurality of partial depth ranges in a depth range of an eye to be inspected, the fourth image having at least one of lower noise and higher contrast compared to the third image, wherein the plurality of partial depth ranges are different from each other, and Here, an image processing device is used which is configured to be able to determine a depth range different from the plurality of partial depth ranges different from each other as the at least partial depth range.

22. A storage medium storing a program, wherein when the program is executed by a processor, the processor executes each step of the medical image processing method according to claim 21.

Images