Image processing device, image processing method, and computer readable medium

Abstract

The present invention relates to an image processing device, an image processing method and a computer-readable medium. An image processing device is provided, comprising: an acquisition unit configured to acquire a first tomographic image of an object; and an image quality improvement unit configured to generate a second tomographic image from the first tomographic image, the second tomographic image appearing to be an image in which different image processing has been performed on different regions in the first tomographic image using a learning model.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a computer-readable medium. Background Art

[0002] Various ophthalmic devices using optical devices are currently being used. For example, various devices such as anterior segment imaging devices, fundus cameras, and scanning laser ophthalmoscopy (SLO) are being used as optical devices for observing eyes.

[0003] Among such devices, an optical coherence tomography device (OCT device) based on optical coherence tomography (OCT) utilizing multi-wavelength light wave interference is capable of obtaining a tomographic image of a sample with high resolution. Therefore, in clinics specializing in retinal care, OCT devices are indispensable ophthalmic equipment. OCT devices are also being used in endoscopes, etc., not only in ophthalmology. OCT devices are widely used in ophthalmic diagnosis, etc., to obtain a tomographic image of the retina of the fundus of the eye to be examined or a tomographic image of the anterior segment of the eye such as the cornea.

[0004] The source data of the tomographic image captured by the OCT device is generally about 32-bit floating point data or 10-bit or more integer data, and is data with a high dynamic range, which includes data ranging from very low luminance information to high luminance information. On the other hand, the data that can be displayed on a general display is data with a relatively low dynamic range, such as 8-bit integer data. Therefore, when such source data with a high dynamic range is converted as is into data with a low dynamic range for display, the contrast of the retina part, which is important for the diagnosis of the fundus of the eye, is significantly reduced.

[0005] Therefore, in a general OCT apparatus, when converting source data of a tomographic image into data for display, a certain amount of low-luminance side data is discarded to obtain good contrast of the retinal part. In this case, in the displayed tomographic image, the contrast of the region related to the vitreous body or the choroidal part, etc. displayed as a low-luminance region is reduced, and it becomes difficult to observe the internal structure of the vitreous body or the choroidal part.

[0006] On the other hand, when the source data of the tomographic image is grayscale converted to ensure the contrast of areas related to the vitreous body or the choroid so as to more closely observe the internal structure of the vitreous body or the choroid, the contrast of the high-brightness retinal area is reduced and it becomes difficult to observe the retinal area.

[0007] In recent years, it has become necessary to observe not only a part of the eye but also a whole part of the eye. In view of this need, Patent Document 1 proposes a method for segmenting a tomographic image, setting display conditions for each identified local area, and performing grayscale conversion processing.

[0008] [Citation List]

[0009] [Patent Document]

[0010] Patent Document 1: International Publication No. WO 2014 / 203901 Summary of the invention

[0011] [Technical issues]

[0012] In a diseased eye, the shape of the retina becomes irregular due to the disappearance of layers, bleeding, and the appearance of leukocytes or new blood vessel sites. Therefore, in the conventional segmentation processing method that uses the regularity of the retinal shape to determine the image feature extraction result and detect the boundary of the retinal layer, there are limitations such as false detection when the boundary of the retinal layer is automatically detected. In this case, due to false detection in the segmentation processing, etc., there is a case where grayscale conversion processing, etc. cannot be appropriately performed on each local area (observation target) in order to fully observe the eye to be inspected.

[0013] Therefore, an object of the present invention is to provide an image processing apparatus, an image processing method, and a computer-readable medium storing a program, which are capable of generating an image that appears to be an image to which image processing suitable for each region of an observation target has been performed.

[0014] [Solution to the problem]

[0015] According to one embodiment of the present invention, an image processing device includes: an acquisition unit, which is configured to obtain a first medical image of a subject; and an image quality improvement unit, which is configured to generate a second medical image from the first medical image, wherein the second medical image appears to be an image in which different image processing has been performed on different regions in the first medical image using a learning model.

[0016] According to another embodiment of the present invention, an image processing method includes: obtaining a first medical image of a subject; and generating a second medical image from the first medical image, wherein the second medical image appears to be an image in which different image processing has been performed on different regions in the first medical image using a learning model.

[0017] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] [Figure 1 ] Figure 1 : is a diagram showing a schematic configuration example of an OCT apparatus according to Embodiment 1.

[0019] [ Figure 2 ] Figure 2 is a diagram showing a schematic configuration example of an image pickup unit according to Embodiment 1.

[0020] [ Figure 3 ] Figure 3 is a diagram showing a schematic configuration example of a control unit according to Embodiment 1.

[0021] [ Figure 4 ] Figure 4 It is an explanatory diagram for describing the division of the retinal part, the vitreous body part, and the choroidal part.

[0022] [ Figure 5A ] Figure 5A is an explanatory diagram for describing a general display image process.

[0023] [ Figure 5B ] Figure 5B is an explanatory diagram for describing a general display image process.

[0024] [ Figure 5C ] Figure 5C is an explanatory diagram for describing a general display image process.

[0025] [ Figure 5D ] Figure 5D is an explanatory diagram for describing a general display image process.

[0026] [ Fig. 6A ] Fig. 6A This is an explanatory diagram for describing conversion processing to facilitate observation of the retina.

[0027] [ Figure 6B ] Figure 6B This is an explanatory diagram for describing conversion processing to facilitate observation of the retina.

[0028] [ Fig. 7A ] Fig. 7A This is an explanatory diagram used to describe the conversion process that facilitates observation of the vitreous body and the choroid.

[0029] [ Figure 7B ] Figure 7B This is an explanatory diagram used to describe the conversion process that facilitates observation of the vitreous body and the choroid.

[0030] [ Fig. 8A ] Fig. 8A is a diagram showing an example of training data.

[0031] [ Figure 8B ] Figure 8B is a diagram showing an example of training data.

[0032] [ Fig.9A ] Fig.9A is a diagram showing an example of training data.

[0033] [ Fig. 9B ] Fig. 9B is a diagram showing an example of training data.

[0034] [ Fig.10 ] Fig.10 is a diagram showing an example of the construction of a learning model.

[0035] [ Fig.11 ] Fig.11 is a flowchart showing a series of image processing operations according to Embodiment 1.

[0036] [ Fig. 12A ] Fig. 12A is a diagram showing different examples of training data.

[0037] [ Fig. 12B ] Fig. 12B is a diagram showing different examples of training data.

[0038] [ Fig. 12C ] Fig. 12C is a diagram showing different examples of training data.

[0039] [ Fig.13A ] Fig.13A It is an explanatory diagram for describing imaging in the vitreous body mode.

[0040] [ Fig. 13B ] Fig. 13B It is an explanatory diagram for describing imaging in the vitreous body mode.

[0041] [ Fig. 13C ] Fig. 13C It is an explanatory diagram for describing imaging in the vitreous body mode.

[0042] [ Fig.14A ] Fig.14A This is an explanatory diagram for describing imaging in the choroidal mode.

[0043] [ Fig. 14B ] Fig. 14B This is an explanatory diagram for describing imaging in the choroidal mode.

[0044] [ Fig. 14C ] Fig. 14C This is an explanatory diagram for describing imaging in the choroidal mode.

[0045] [ Fig.15A ] Fig.15A is a diagram showing different examples of training data.

[0046] [ Fig. 15B ] Fig. 15B is a diagram showing different examples of training data.

[0047] [ Fig. 15C ] Fig. 15C is a diagram showing different examples of training data.

[0048] [ Fig.16 ] Fig.16 is a diagram showing a schematic configuration example of a control unit according to Embodiment 2.

[0049] [ Fig.17 ] Fig.17 is a flowchart showing a series of image processing operations according to Embodiment 2.

[0050] [ Fig.18A ] Fig.18A : is a diagram showing an example of a display screen for selecting a desired area of ​​attention.

[0051] [ Fig.18B ] Fig.18B : is a diagram showing an example of a display screen for selecting a desired area of ​​attention.

[0052] [ Fig.18C ] Fig.18C : is a diagram showing an example of a display screen for selecting a desired area of ​​attention.

[0053] [ Fig.19 ] Fig.19 is a diagram showing a schematic configuration example of a control unit according to Embodiment 3.

[0054] [ Fig. 20 ] Fig. 20 is a flowchart showing a series of image processing operations according to Embodiment 3.

[0055] [ Fig.21A ] Fig.21A is a diagram showing an example of a plurality of OCTA en-face images.

[0056] [ Fig.21B ] Fig.21B is a diagram showing an example of a plurality of tomographic images.

[0057] [ Fig.22A ] Fig.22A is a diagram showing an example of a user interface according to Embodiment 4.

[0058] [ Fig. 22B ] Fig. 22Bis a diagram showing an example of a user interface according to Embodiment 4.

[0059] [ Fig.23 ] Fig.23 is a diagram showing an example of a user interface according to Embodiment 4.

[0060] [ Fig.24A ] Fig.24A is a diagram showing a configuration example of a neural network used as a machine learning model according to Modification 13.

[0061] [ Fig. 24B ] Fig. 24B is a diagram showing a configuration example of a neural network used as a machine learning model according to Modification 13.

[0062] [ Fig.25A ] Fig.25A is a diagram showing a configuration example of a neural network used as a machine learning model according to Modification 13.

[0063] [ Fig.25B ] Fig.25B is a diagram showing a configuration example of a neural network used as a machine learning model according to Modification 13. DETAILED DESCRIPTION

[0064] The exemplary embodiments for implementing the present invention will now be described in detail with reference to the accompanying drawings. However, the size, material, shape and relative position of the components described in the following embodiments are not determined and may be changed according to the configuration of the device to which the present invention is applied or according to various conditions. In addition, in different drawings, the same or functionally similar elements are represented by the same reference numerals.

[0065] Hereinafter, the term "machine learning model" refers to a learning model learned according to a machine learning algorithm. Specific examples of algorithms for machine learning include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning (deep structured learning) that generates feature values ​​using the neural network itself and combines weighting factors for learning can also be mentioned. The algorithms that can be used in the above algorithms can be appropriately used and applied to the embodiments and variations described below. In addition, the term "training data" refers to training data consisting of paired input data and output data. In addition, the term "correct answer data" refers to output data of training data.

[0066] Note that the term "learning model" refers to a model that has been trained (learned) in advance using appropriate training data for a machine learning model according to any machine learning algorithm such as deep learning. However, although the learning model is a model obtained in advance using appropriate training data, it is assumed that the learning model is not a model that does not perform further learning, but a model that can perform incremental learning. Incremental learning can also be performed after the device is installed at the destination of use.

[0067] (Example 1)

[0068] Now refer to Figures 1 to 13C An OCT apparatus according to Embodiment 1 is described. Figure 1 is a diagram showing a schematic configuration example of an OCT apparatus according to the present embodiment.

[0069] (Main structure)

[0070] The OCT device 1 includes an imaging unit 20, a control unit 30 (image processing device), an input unit 40, and a display unit 50. A measuring optical system 21, a stage unit 25, and a base unit 23 are provided in the imaging unit 20. The measuring optical system 21 can obtain an anterior segment image, an SLO fundus image of the subject's eye, and a tomographic image. The measuring optical system 21 is provided on the base unit 23 through the stage unit 25. The stage unit 25 supports the measuring optical system 21 so as to be movable in the front, rear, left, and right directions. A spectrometer, etc., which will be described later, is provided in the base unit 23.

[0071] The control unit 30 is connected to the imaging unit 20 and the display unit 50, and can control these units. The control unit 30 can generate a tomographic image based on the tomographic information obtained from the imaging unit 20, etc. and perform image processing, etc. Note that the control unit 30 can be connected to any other device not shown in the figure through any network such as the Internet.

[0072] The input unit 40 is also connected to the control unit 30. The input unit 40 is operated by the operator (examiner), and is used to input instructions to the control unit 30. The input unit 40 may include any input device, and, for example, may include a keyboard and a mouse, etc. The display unit 50 is composed of any display, and can display information related to the subject and various images, etc. according to the control of the control unit 30.

[0073] (Structure of the camera unit)

[0074] Next, we will refer to Figure 2 The configuration of the image pickup unit 20 is described. Figure 2 is a diagram showing a schematic configuration example of the image pickup unit 20 according to the present embodiment.

[0075] First, the configuration of the measuring optical system 21 will be described. In the measuring optical system 21, an objective lens 201 is arranged to face the subject's eye E, and a first dichroic mirror 202 and a second dichroic mirror 203 are arranged on the optical axis of the objective lens 201. The optical path from the objective lens 201 is branched by the dichroic mirrors into an optical path L1 of the OCT optical system, an optical path L2 for a fixation lamp and an SLO optical system according to corresponding wavelength bands, and an optical path L3 for observing the anterior segment of the eye, the SLO optical system being used to observe the subject's eye E and acquire an SLO fundus image.

[0076] Note that in the present embodiment, the optical path L3 for observing the anterior segment of the eye is set in the reflection direction of the first dichroic mirror 202, and the optical path L1 of the OCT optical system and the optical path L2 for the SLO optical system and the fixation lamp are set in the transmission direction of the first dichroic mirror 202. In addition, the optical path L1 of the OCT optical system is set in the reflection direction of the second dichroic mirror 203, and the optical path L2 for the SLO optical system and the fixation lamp is set in the transmission direction of the second dichroic mirror 203. However, the directions in which the optical paths of the respective optical systems are set are not limited thereto, and can be arbitrarily changed according to the desired configuration.

[0077] The SLO scanning unit 204, lenses 205 and 206, a reflecting mirror 207, a third dichroic mirror 208, a photodiode 209, an SLO light source 210, and a fixation lamp 211 are arranged on an optical path L2 for the SLO optical system and the fixation lamp. Note that in the present embodiment, the SLO light source 210 is arranged in the reflection direction of the third dichroic mirror 208, and the fixation lamp 211 is arranged in the transmission direction of the third dichroic mirror 208. However, the fixation lamp 211 may be arranged in the reflection direction of the third dichroic mirror 208, and the SLO light source 210 may be arranged in the transmission direction of the third dichroic mirror 208.

[0078] The SLO scanning unit 204 is a scanning unit that scans light emitted from the SLO light source 210 and the fixation lamp 211 above the subject's eye E, and includes an X-scanner that scans in the X-axis direction and a Y-scanner that scans in the Y-axis direction. In the present embodiment, since the X-scanner is required to perform high-speed scanning, the X-scanner is composed of a polygon mirror, and the Y-scanner is composed of a galvanometer mirror. Note that the configuration of the SLO scanning unit 204 is not limited to this configuration, and can be arbitrarily changed according to a desired configuration.

[0079] The lens 205 can be driven along the optical axis direction indicated by the arrow in the figure by a motor (not shown) or the like controlled by the control unit 30 to focus the SLO optical system and the fixation lamp. The reflector 207 is a prism on which a perforated reflector or a hollow reflector has been deposited, and can separate the projection light from the SLO light source 210 and the return light from the subject's eye E. The third dichroic mirror 208 separates the optical path to the SLO light source 210 and the optical path to the fixation lamp 211 for each wavelength band.

[0080] For example, the SLO light source 210 generates light having a wavelength of around 780 nm. The photodiode 209 detects return light from the subject's eye E for the projection light irradiated from the SLO light source 210. The fixation lamp 211 is used to generate visible light to promote visual fixation of the subject.

[0081] The projection light emitted from the SLO light source 210 is reflected by the third dichroic mirror 208, transmitted through the reflection mirror 207, passes through the lenses 206 and 205, and is scanned by the SLO scanning unit 204 over the subject's eye E. The return light from the subject's eye E returns along the same path as the projection light, and then is reflected by the reflection mirror 207 and guided to the photodiode 209. The control unit 30 can generate an SLO fundus image based on the drive position of the SLO scanning unit 204 and the output of the photodiode 209.

[0082] The light emitted from the fixation lamp 211 is transmitted through the third dichroic mirror 208 and the reflecting mirror 207, passes through the lenses 206 and 205, and is scanned by the SLO scanning unit 204 over the subject's eye E. At this time, by blinking the fixation lamp 211 according to the movement of the SLO scanning unit 204, the control unit 30 can form an arbitrary shape at any position on the subject's eye E, thereby promoting visual fixation of the subject.

[0083] Lenses 212 and 213, a separation prism 214, and a CCD 215 for observing the anterior segment of the eye of the eye for detecting infrared light are arranged on an optical path L3 for observing the anterior segment of the eye of the eye. The CCD 215 has sensitivity to the wavelength of the irradiation light for observing the anterior segment of the eye of the eye (not shown), particularly to wavelengths near 970 nm. The separation prism 214 is arranged at a position conjugate with the pupil of the eye E to be inspected. The control unit 30 can generate an anterior segment image based on the output of the CCD 215. By using the separation image of the anterior segment of the eye of the eye based on the light passing through the separation prism 214, the control unit 30 can detect the distance of the measuring optical system 21 relative to the eye E to be inspected in the Z-axis direction (front-back direction).

[0084] An OCT optical system for capturing a tomographic image of the eye E to be inspected is disposed on the optical path L1 of the OCT optical system. More specifically, the OCT optical system is used to obtain an interference signal to generate a tomographic image of the eye E to be inspected.

[0085] An XY scanner 216, a lens 217, a lens 218, and a fiber end of an optical fiber 224 are provided on the optical path L1 of the OCT optical system. The XY scanner 216 is an OCT scanning unit for scanning the measurement light described later above the eye E to be inspected. Although the XY scanner 216 is shown as a single mirror, it is composed of two galvanometer mirrors for scanning the measurement light in two directions, namely, the X-axis direction and the Y-axis direction. Note that the construction of the XY scanner 216 is not limited to this construction, and can be arbitrarily changed according to the desired construction. For example, the XY scanner 216 can be composed of a MEMS mirror capable of deflecting light in two-dimensional directions using one mirror.

[0086] The lens 217 can be driven in the optical axis direction indicated by the arrow in the figure by a motor (not shown) or the like controlled by the control unit 30. By driving the lens 217 using a motor (not shown) or the like, the control unit 30 can focus the measurement light emitted from the optical fiber 224 connected to the optical coupler 219 onto the eye to be inspected E. As a result of this focusing, the return light of the measurement light from the eye to be inspected E is imaged as a point at the tip of the optical fiber 224 and simultaneously enters the optical fiber 224.

[0087] Next, the configuration of the optical path extending from the OCT light source 220, the reference optical system, and the spectrometer 230 will be described. The OCT light source 220 is connected to the optical coupler 219 through the optical fiber 225. Optical fibers 224, 225, 226, and 227 are connected to the optical coupler 219. The optical fibers 224, 225, 226, and 227 are single-mode optical fibers integrally connected to the optical coupler 219.

[0088] The fiber end of the optical fiber 224 is set on the OCT optical path L1, and the measurement light passes through the optical fiber 224 and the polarization adjustment unit 228 provided on the measurement light side of the optical fiber 224, thereby entering the OCT optical path L1. On the other hand, the fiber end of the optical fiber 226 is set on the optical path of the reference optical system, and the reference light described later passes through the optical fiber 226 and the polarization adjustment unit 229 provided on the reference light side of the optical fiber 226, thereby entering the optical path of the reference optical system. A lens 223, a dispersion compensation glass 222, and a reference reflector 221 are provided on the optical path of the reference optical system. A spectrometer 230 is connected to the optical fiber 227.

[0089] These components constitute a Michelson interferometer system. Note that although the Michelson interferometer system is used as the interferometer system in this embodiment, a Mach-Zehnder interferometer system may also be used. Depending on the light quantity difference between the measurement light and the reference light, the Mach-Zehnder interferometer system may be used when the light quantity difference is large, and the Michelson interferometer system may be used when the light quantity difference is relatively small.

[0090] The OCT light source 220 emits light to be used for measurement by the OCT. In the present embodiment, a superluminescent diode (SLD), which is a typical low-coherence light source, is used as the OCT light source 220. The SLD used in the present embodiment has a central wavelength of 855 nm and a wavelength bandwidth of about 100 nm. In this case, the bandwidth affects the optical axis resolution of the obtained tomographic image and is therefore an important parameter. In addition, as for the type of light source, although SLD is selected in this case, it is sufficient as long as the light source can emit low-coherence light and amplified spontaneous emission (ASE) and the like can also be used. Considering the purpose of photographing the eye, the central wavelength can fall within the near-infrared range. In addition, since the central wavelength affects the lateral resolution of the obtained tomographic image, the central wavelength can be made as short as possible. In the present embodiment, due to the above two reasons, the central wavelength is set to 855 nm.

[0091] The light emitted from the OCT light source 220 enters the optical coupler 219 through the optical fiber 225. The light entering the optical coupler 219 is divided by the optical coupler 219 into the measurement light traveling to the optical fiber 224 side and the reference light traveling to the optical fiber 226 side. The measurement light passes along the above-mentioned optical path L1 of the OCT optical system and is irradiated onto the subject's eye E as the subject. The return light of the measurement light generated as a result of the reflection and scattering of the measurement light by the subject's eye E travels along the same optical path as the measurement light and reaches the optical coupler 219.

[0092] On the other hand, the reference light reaches the reference reflector 221 via the optical fiber 226, the lens 223 and the dispersion compensation glass 222, which is inserted to match the dispersion of the measurement light with the dispersion of the reference light, and then is reflected by the reference reflector 221. Thereafter, the reference light returns along the same optical path and reaches the optical coupler 219. The reference reflector 221 is adjustable by a motor (not shown) controlled by the control unit 30 or the like in the direction of the optical axis indicated by the arrow in the figure.

[0093] The measuring light and the reference light are coupled together to become interference light by the optical coupler 219. In this case, interference occurs when the optical path length of the measuring light and the optical path length of the reference light are substantially equal to each other. The control unit 30 can match the optical path length of the reference light with the optical path length of the measuring light that varies according to the eye E to be inspected by controlling a motor (not shown) or the like to move the reference reflector 221 in the optical axis direction.

[0094] Note that the polarization adjustment unit 228 on the measurement light side and the polarization adjustment unit 229 on the reference light side each have a plurality of sections of optical fiber loops. By rotating the loop sections around the longitudinal direction of each optical fiber to twist the relevant optical fiber, the polarization adjustment units 228 and 229 can adjust and match the polarization states of the measurement light and the reference light.

[0095] The interference light generated at the optical coupler 219 is guided to the spectrometer 230 provided in the base unit 23 via the optical fiber 227. Lenses 234 and 232, a diffraction grating 233, and a line sensor 231 are provided in the spectrometer 230. The interference light emitted from the optical fiber 227 is converted into parallel light by the lens 234, then spectrally separated by the diffraction grating 233, and imaged on the line sensor 231 by the lens 232. The control unit 30 can generate a tomographic image of the subject's eye E using an interference signal based on the interference light output from the line sensor 231.

[0096] With the above configuration, a tomographic image of the subject's eye E can be obtained using the imaging unit 20 , and even in the case of near-infrared light, an SLO fundus image of the subject's eye E with high contrast can be obtained.

[0097] (Method of Capturing Tomographic Images)

[0098] Next, a method of capturing a tomographic image using the OCT device 1 will be described. In the OCT device 1, by controlling the XY scanner 216 by the control unit 30, a tomographic image of a predetermined portion of the eye E to be inspected can be captured. Here, the trajectory along which the measuring light scans over the eye E to be inspected is referred to as a "scanning pattern". Examples of scanning patterns include: a cross scan in which a cross shape centered on a point is scanned in the vertical and horizontal directions; and a 3D scan in which a three-dimensional tomographic image is obtained as a result of scanning to fill the entire area. Cross scanning is suitable for situations where detailed observation of a specific portion is desired, while 3D scanning is suitable for situations where the layer structure or layer thickness of the entire retina is desired to be observed.

[0099] Here, the imaging method in the case of performing 3D scanning will be described. First, the measuring light is scanned in the X-axis direction (main scanning direction) in the figure, and the line sensor 231 obtains information related to the predetermined number of rounds of imaging from the imaging range in the X-axis direction of the subject's eye E.

[0100] Here, obtaining tomographic information in the depth direction at a point in the X-axis direction of the subject's eye E is referred to as an "A scan". The luminance distribution on the line sensor 231 obtained by the A scan is subjected to a fast Fourier transform (FFT), and in order to display the obtained information on the display unit 50, the linear luminance distribution obtained by the FFT is converted into density information. Thus, an A scan image based on the information obtained by the A scan can be generated. In addition, by aligning a plurality of A scan images, a B scan image as a two-dimensional image can be obtained.

[0101] Multiple B-scan images can be obtained by photographing multiple A-scan images to form a B-scan image, then moving the scanning position in the Y-axis direction (sub-scanning direction) and scanning again in the X-axis direction. By displaying multiple B-scan images or a three-dimensional tomographic image composed of multiple B-scan images on the display unit 50, the inspector can observe the three-dimensional tomographic image state of the eye E to be inspected. The inspector can diagnose the eye E to be inspected based on the image. Although an example in which a three-dimensional tomographic image is obtained by obtaining multiple B-scan images in the X-axis direction has been described here, a three-dimensional tomographic image can also be obtained by obtaining multiple B-scan images in the Y-axis direction. Please note that the scanning direction is not limited to the X-axis direction and the Y-axis direction, and it is sufficient that the scanning direction is an axial direction that is orthogonal to the Z-axis direction and intersects with each other.

[0102] (Structure of control unit)

[0103] Next, refer to Figure 3 The control unit 30 is described. Figure 3 2 shows a schematic configuration example of the control unit 30 . An obtaining unit 310 , an image processing unit 320 , a driving control unit 330 , a memory 340 , and a display control unit 350 are provided in the control unit 30 .

[0104] The obtaining unit 310 can obtain data of output signals of the CCD 215 and the photodiode 209, and data of an output signal of the line sensor 231 corresponding to the interference signal of the subject's eye E from the imaging unit 20. Note that the data of the output signal obtained by the obtaining unit 310 may be an analog signal or a digital signal. In the case where the obtaining unit 310 obtains an analog signal, the control unit 30 may convert the analog signal into a digital signal.

[0105] In addition, the acquisition unit 310 can obtain various data such as tomographic data generated by the image processing unit 320, and various images such as tomographic images, SLO fundus images, and anterior segment images. Here, the term "tomographic data" refers to data including information related to a cross section of the object to be inspected, and includes a signal obtained by Fourier transforming an interference signal obtained by OCT, a signal obtained by performing any processing on the relevant signal, and the like.

[0106] In addition, the acquisition unit 310 can obtain a shooting condition group of the image to be image-processed (for example, information related to the shooting date and time, the shooting part name, the shooting area, the shooting angle of view, the shooting system, the image resolution and grayscale, the image size, the image filter, and the image data format). Please note that the shooting condition group is not limited to the examples of the aforementioned shooting condition group. In addition, the shooting condition group does not have to include all the conditions mentioned in the aforementioned examples, and may include some of these conditions.

[0107] Specifically, the acquisition unit 310 acquires the imaging condition of the imaging unit 20 when the relevant image is imaged. In addition, depending on the data format of the image, the acquisition unit 310 may also acquire the imaging condition group stored in the data structure constituting the image. Note that, in the case where the imaging condition is not stored in the data structure of the image, the acquisition unit 310 may also separately acquire the imaging information group including the imaging condition group from a storage device or the like storing the imaging condition.

[0108] In addition, the obtaining unit 310 may also obtain information for identifying the subject's eye, such as a subject identification number, from the input unit 40, etc. Note that the obtaining unit 310 may obtain various data, various images, or various information from the memory 340 or another device (not shown) connected to the control unit 30. The obtaining unit 310 may store the obtained various data or images in the memory 340.

[0109] The image processing unit 320 may generate a tomographic image from the data obtained by the obtaining unit 310 or the data stored in the memory 340, and may perform image processing on the generated or obtained tomographic image. A tomographic image generating unit 321 and an image quality improving unit 322 are provided in the image processing unit 320.

[0110] The tomographic image generation unit 321 can perform wave number conversion, Fourier transform, absolute value conversion (acquisition of amplitude), etc. on the interference signal data obtained by the obtaining unit 310 to generate tomographic data, and can generate a tomographic image of the subject's eye E based on the tomographic data. The interference signal data obtained by the obtaining unit 310 may be data of a signal output from the line sensor 231, or may be data of an interference signal obtained from the memory 340 or a device (not shown) connected to the control unit 30. Note that any known method may be adopted as a method for generating a tomographic image, and a detailed description thereof is omitted here.

[0111] The image quality improving unit 322 generates a high-quality tomographic image based on the tomographic image generated by the tomographic image generating unit 321, using a learning model described later. Note that the image quality improving unit 322 can also generate a high-quality tomographic image based on the tomographic image obtained by the obtaining unit 310 from the memory 340 or another device (not shown) connected to the control unit 30, not just based on the tomographic image captured using the imaging unit 20 or the like.

[0112] The driving control unit 330 can control the driving of components of the imaging unit 20 connected to the control unit 30, such as the OCT light source 220, the XY scanner 216, the lens 217, the reference mirror 221, the SLO light source 210, the SLO scanning unit 204, the lens 205 and the fixation lamp 211.

[0113] The memory 340 can store various data obtained by the acquisition unit 310, as well as various data and images such as tomographic images generated and processed by the image processing unit 320. In addition, the memory 340 can store the attributes of the subject (name, age, etc.), information related to the subject's eye such as measurement results obtained using other inspection equipment (axial length of the eyeball, intraocular pressure, etc.), imaging parameters, image analysis parameters, and parameters set by the operator. In addition, the memory 340 can also store statistical information of a general database. Please note that a structure in which these images and information are stored in an external storage device (not shown) can also be adopted. The memory 340 can also store programs executed by the processor to implement the functions of the various components of the control unit 30.

[0114] The display control unit 350 can display various information obtained by the obtaining unit 310 and various images such as tomographic images generated and processed by the image processing unit 320 on the display unit 50. The display control unit 350 can also display information input by a user or the like on the display unit 50.

[0115] The control unit 30 can be constituted, for example, by using a general-purpose computer. Note that the control unit 30 can be constituted by using a dedicated computer of the OCT device 1. The control unit 30 is equipped with a CPU (Central Processing Unit) (not shown) or an MPU (Micro Processing Unit) and a storage medium including a memory such as an optical disk or a ROM (Read Only Memory). Each component other than the memory 340 of the control unit 30 can be constituted by a software module executed by a processor such as a CPU or an MPU. In addition, the various components in question can be constituted by a circuit having a specific function such as an ASIC or an independent device. For example, the memory 340 can be constituted by any storage medium such as an optical disk or a memory.

[0116] Note that the control unit 30 may include one or more processors such as a CPU and a storage medium such as a ROM. Therefore, the components of the control unit 30 may be configured to function when at least one processor and at least one storage medium are connected and at least one processor executes a program stored in at least one storage medium. Note that the processor is not limited to a CPU or an MPU, and may be a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array), etc.

[0117] Next, in order to describe the image quality improvement processing for tomographic images according to the present embodiment, reference will be made to Figures 4 to 5D Describe the segmentation process and the grayscale conversion process.

[0118] (Division Processing)

[0119] Figure 4 is a diagram showing an example of a tomographic image in which the boundaries of the respective regions of the retinal layer are detected by segmentation processing. The segmentation processing of the tomographic image can detect the boundaries of the respective regions included in the tomographic image. Figure 4 In the case of the tomographic image 400 shown, a boundary 401 between the vitreous body and the retinal body and a boundary 402 between the retinal body and the choroidal body are detected. By detecting the boundaries 401 and 402 in the tomographic image 400, a region 403 of the retinal body between the boundary 401 and the boundary 402, a region 404 of the vitreous body on the superficial side from the boundary 401, and a region 405 of the choroidal body on the deep side from the boundary 402 can be identified.

[0120] Any known method can be used as the segmentation process. As an example, first, a median filter and a Sobel filter are applied to the tomographic image as the object to be processed, respectively, thereby generating a median image and a Sobel image. Next, from the generated median image and Sobel image, contours are generated for each item of the tomographic data corresponding to the A scan. Here, the generated contour is a brightness value contour in the case of the intermediate image and a gradient contour in the case of the Sobel image. Thereafter, a peak value in the contour generated from the Sobel image is detected. By referring to the median image contour corresponding to the previous and subsequent areas or the area between the peaks of the detected peak, the boundaries between the various areas in the retinal layer can be detected.

[0121] (Grayscale conversion processing)

[0122] Next, refer to FIG. 5A to FIG. 5D , a grayscale conversion process for enhancing the contrast in the region 404 of the vitreous body portion and the region 405 of the choroid portion, the region 403 of the retinal portion, or all of these regions will be described. Figure 5A A tomographic image 500 is shown as an example of a source tomographic image (hereinafter referred to as an "original tomographic image") obtained by photographing the subject's eye E. The tomographic image 500 is generally in an integer format of 10 bits or more, and is data with a high dynamic range, including data ranging from extremely low luminance information to high luminance information. On the other hand, as described above, the data that can be displayed on the display unit 50 is low dynamic range data, such as 8-bit integer data. Therefore, grayscale conversion processing is performed so that the data of the original tomographic image 500 becomes low dynamic range data for display.

[0123] Figure 5B 5 is a diagram showing a tomographic image 501 obtained by performing grayscale conversion processing on an original tomographic image 500 so as to make the region of the retinal part easy to observe, in other words, to ensure the contrast of the region of the retinal part. Fig. 6A and Figure 6B A grayscale conversion process for ensuring contrast in the retinal region is described.

[0124] Fig. 6A 5 is a view showing the frequency of occurrence of luminance values ​​in the tomographic image 500, and showing a range 601 of luminance values ​​corresponding to the luminance values ​​of the region of the retina. Note that the range of luminance values ​​corresponding to the luminance values ​​of the region of the retina can be determined based on an average luminance range empirically obtained for the region of the retina. In the grayscale conversion processing discussed, the conversion processing is performed so that the luminance range 601 corresponding to the luminance values ​​of the region of the retina becomes a wide range of luminance values ​​related to the data for display, such as Figure 6B In this way, a tomographic image 501 for display in which the retinal region is easily observable can be generated.

[0125] Figure 5C 5 is a diagram showing a tomographic image 502 obtained by performing grayscale conversion processing on the original tomographic image 500 so that the regions of the vitreous body and the choroidal portion are easy to observe, in other words, ensuring the contrast of the regions of the vitreous body and the choroidal portion. Fig. 7A and Figure 7B A grayscale conversion process for ensuring contrast in the vitreous and choroidal regions is described.

[0126] Fig. 7A 701 is a view showing the frequency of occurrence of luminance values ​​in the tomographic image 500, and showing a range 701 of luminance values ​​corresponding to the luminance values ​​of the regions of the vitreous body and the choroid. Note that the range of luminance values ​​corresponding to the luminance values ​​of the regions of the vitreous body and the choroid can be determined based on an average luminance range empirically obtained for the regions of the vitreous body and the choroid. In the grayscale conversion processing discussed, the conversion processing is performed so that the luminance range 701 corresponding to the luminance values ​​of the regions of the vitreous body and the choroid becomes a wide range of luminance values ​​related to the data for display, such as Figure 7B In this way, a tomographic image 502 for display can be generated in which the vitreous body and choroid regions are easily observable.

[0127] Figure 5D5 is a view showing a tomographic image 503 obtained by performing grayscale conversion processing so that the regions of the retina, vitreous body, and choroid are easily observed, in other words, so that the contrast of these regions is ensured. In this case, first, the boundary 401 between the vitreous body and the retina and the boundary 402 between the retina and the choroid are detected by the above-mentioned segmentation processing, and the region 403 of the retina, the region 404 of the vitreous body, and the region 405 of the choroid are identified.

[0128] Thereafter, the retinal region 403 is subjected to grayscale conversion processing, so that the range 601 of luminance values ​​corresponding to the retinal region becomes a wide range of luminance values ​​related to the data for display, such as Figure 6B On the other hand, the vitreous region 404 and the choroid region 405 are subjected to grayscale conversion processing, so that the range 701 of luminance values ​​corresponding to the vitreous region and the choroid region becomes a wide range of luminance values ​​related to the data for display, as shown in FIG. Figure 7B In this way, a tomographic image 503 for display in which the regions of the retina, vitreous body, and choroid are easily observable can be generated.

[0129] Note that not only the same conversion processing can be performed on the vitreous body and the choroid, but also different conversion processing can be performed between the vitreous body and the choroid. In addition, not only linear conversion processing can be performed, but also S-curve conversion processing such as S-shaped conversion or γ-conversion can be performed.

[0130] In the grayscale conversion process for generating a tomographic image in which the entire area of ​​the retina, vitreous body, and choroid is easily observed as described above, detection of the area in the tomographic image is performed by segmentation processing. Therefore, if erroneous detection by the segmentation processing occurs due to changes in the layer structure caused by a lesion in the diseased eye, the grayscale conversion process will not be correctly performed in some cases, and a tomographic image in which the entire area is easily observed cannot be generated.

[0131] In contrast, in the case of the control unit 30 according to the present embodiment, a high-quality tomographic image is generated by using a learning model of a machine learning model according to an arbitrary machine learning algorithm such as deep learning, which is easy to observe and appears to be an image in which different image processing has been performed for each region in the tomographic image. When segmentation processing is performed using the learning model, for example, even if the layer structure changes due to a lesion in a diseased eye, image processing can be appropriately performed according to the learning trend.

[0132] Note that in this specification, the term "high-quality image" refers to an image that has been converted to have an image quality that is more suitable for image diagnosis, and the term "image quality improvement processing" refers to converting an input image into an image with an image quality that is more suitable for image diagnosis. In this case, the content of the image quality suitable for image diagnosis depends on what kind of diagnosis is expected to be performed using various image diagnoses. Therefore, although it is not possible to say this unconditionally, for example, image quality suitable for image diagnosis includes image quality such as: displaying an image target in a color and grayscale that makes the image target easy to observe; low noise amount, high contrast, large image size, and high resolution. In addition, image quality suitable for image diagnosis may include image quality such that objects or grayscales that do not actually exist and are rendered during the image generation process are removed from the image.

[0133] (Learning about machine learning)

[0134] Now refer to FIG. 8A to FIG. 10 Describe the learning model according to this embodiment. First, refer to FIG. 8A to FIG. 9B Describes the training data associated with the learned model.

[0135] The training data is composed of one or more pairs of input data and output data. In the present embodiment, specifically, the training data is composed of pairs in which an original tomographic image obtained by an OCT apparatus such as tomographic image 500 is used as input data, and a tomographic image that has been image-processed so as to be fully observed such as tomographic image 503 is used as output data. Note that an image obtained by image processing a tomographic image used as input data may be used as output data.

[0136] First, we describe a case where a pair of pairs constituting the training data is considered to be composed of Fig. 8A and Figure 8B The original tomographic image 810 and the high-quality tomographic image 820 shown in FIG. Fig. 8A and Figure 8B As shown in FIG. 8 , a pair is formed in which the entire original tomographic image 810 is used as input data and the entire high-quality tomographic image 820 is used as output data. Note that although Fig. 8A and Figure 8B In the example shown in , a pair consisting of input data and output data is formed by using each image as a whole, but a pair is not limited to this.

[0137] For example, Fig.9A and Fig. 9BAs shown in , a pair can be formed in which a rectangular area image 911 in the original tomographic image 910 is adopted as input data, and a rectangular area image 921 as a corresponding imaging area in the high-quality tomographic image 920 is adopted as output data. Here, the rectangular area image 911 and the rectangular area image 921 are images having a corresponding positional relationship in the tomographic image 910 and the high-quality tomographic image 920.

[0138] Note that when learning, the scan range (camera angle of view) and scan density (number of A scans and B scans) can be normalized to make the image size uniform, so that the rectangular area size can be uniform when learning. FIG. 8A to FIG. 9B The rectangular area image shown in is an example of the rectangular area sizes when the respective rectangular area sizes are used for learning individually from each other.

[0139] In addition, the number of rectangular regions in Fig. 8A and Figure 8B In the example shown in the Fig.9A and Fig. 9B In the example shown in FIG. 1 , multiple rectangular areas can be set. For example, in Fig.9A and Fig. 9B In the example shown in , it is also possible to constitute a pair in which rectangular area images 912 and 913 in the tomographic image 910 are used as input data, and rectangular area images 922 and 923 as corresponding imaging areas in the high-quality tomographic image 920 are used as output data. Therefore, a pair of rectangular area images different from each other can be created from a pair consisting of one tomographic image and one high-quality tomographic image. Note that the contents of the pair group constituting the training data can be enhanced by creating a large number of rectangular area image pairs while changing the position of the rectangular area to different coordinates in the tomographic image as the source image and the high-quality tomographic image.

[0140] Here, the rectangular area image 911 is an image of the area of ​​the retina portion in the original tomographic image 910, and the rectangular area image 921 is an image of the area of ​​the retina portion in the high-quality tomographic image 920 that has been subjected to image processing such as grayscale conversion processing so as to enable full observation. Similarly, the rectangular area image 912 is an image of the area of ​​the vitreous portion in the original tomographic image 910, and the rectangular area image 922 is an image of the area of ​​the vitreous portion in the high-quality tomographic image 920. Furthermore, the rectangular area image 913 is an image of the area of ​​the choroid portion in the original tomographic image 910, and the rectangular area image 923 is an image of the area of ​​the choroid portion in the high-quality tomographic image 920.

[0141] Despite Fig.9A and Fig. 9BIn the example shown in , the rectangular areas are discretely shown, but both the tomographic image and the high-quality tomographic image as the source image can be divided into a group of rectangular area images with uniform size continuously and without gaps. Alternatively, the tomographic image and the high-quality tomographic image as the source image can be divided into rectangular area image groups at random positions corresponding to each other. In this way, by selecting an image of a smaller area as a pair consisting of input data and output data as a rectangular area, a large amount of pair data can be generated from the tomographic image 910 and the high-quality tomographic image 920 constituting the original pair. Therefore, the time required to train the machine learning model can be shortened.

[0142] Note that the output data is not limited to high-quality tomographic images generated from a single tomographic image. A tomographic image for display that is generated by arithmetic averaging a plurality of tomographic images obtained by imaging the same part of the eye a plurality of times may also be used.

[0143] Note that the rectangular area is not limited to a square and may be a rectangle. In addition, the rectangular area may be an area corresponding to one A scan. Furthermore, when preparing output data for learning, better data can be prepared by making manual adjustments rather than just generating data through predetermined automatic processing.

[0144] In addition, in the pair group constituting the training data, pairs that do not contribute to improving the image quality can be removed from the training data. For example, if the image quality of a high-quality image as output data included in a pair of training data is not suitable for image diagnosis, there is a possibility that an image output by a learning model learned using the relevant training data will have an image quality that is not suitable for image diagnosis. Therefore, by removing pairs whose image quality of output data is not suitable for image diagnosis from the training data, the possibility that the learning model generates an image with an image quality that is not suitable for image diagnosis can be reduced.

[0145] In addition, in the case where the structure or position of the imaging target to be rendered in the image group as a pair is greatly different, there is a possibility that the learning model learned using the relevant training data will output an image that is not suitable for image diagnosis and in which the structure or position of the imaging target is rendered is greatly different from the input image. Therefore, pairs of input data and output data in which the structure or position of the imaging target to be rendered is greatly different between the input data and the output data can also be removed from the training data.

[0146] Next, as an example of a learning model according to this embodiment, using Fig.10 A convolutional neural network (CNN) for image quality improvement of an input tomographic image is described.

[0147] exist Fig.10 The learning model shown in 1001 is composed of a plurality of layer groups responsible for processing a group of input values ​​for output. Note that the types of layers included in the configuration 1001 of the learning model are convolutional layers, downsampling layers, upsampling layers, and pooling layers.

[0148] A convolutional layer is a layer that performs convolution on a set of input values ​​according to set parameters such as the kernel size of the filter, the number of filters, the stride value, and the dilation value. Note that the dimension of the kernel size of the filter can change depending on the dimension of the input image.

[0149] The down-sampling layer is a layer that performs processing such that the number of output value groups is smaller than the number of input value groups by thinning or combining input value groups. Specifically, for example, a maximum pooling process can be used as such a process.

[0150] The up-sampling layer is a layer that performs processing such that the number of output value groups is larger than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. Specifically, for example, linear interpolation processing can be used as such processing.

[0151] The merging layer is a layer that processes by inputting value groups such as an output value group of a certain layer or a pixel value group constituting an image from multiple sources and merging the value groups by concatenating or adding the value groups.

[0152] Please note that as a target included in Fig.10 The parameters set for the convolution layer group in the configuration 1001 shown, for example, by setting the kernel size of the filter to a width of 3 pixels and a height of 3 pixels and setting the number of filters to 64, can perform image quality improvement processing with a certain degree of accuracy. However, it is necessary to be careful in this regard, because if the parameter settings for the layer group and the node group constituting the neural network are different, then in some cases, the degree to which the trend trained based on the training data can be reproduced in the output data will be different. That is, in many cases, the appropriate parameters will be different depending on the form at the time of implementation, so the parameters can be changed to preferred values ​​as needed.

[0153] In addition, there is also the possibility that CNN can obtain better characteristics by changing the construction of CNN rather than just by using the method of changing parameters as described above. The term "better characteristics" refers to, for example, increased accuracy of image quality improvement processing, shorter time spent on image quality improvement processing, and shorter time required to train a machine learning model.

[0154] Note that the construction 1001 of the CNN used in this embodiment is a U-Net type machine learning model, which has the function of an encoder and the function of a decoder, the encoder being composed of multiple layers including multiple downsampling layers, and the decoder being composed of multiple layers including multiple upsampling layers. In the U-Net type machine learning model, position information (spatial information) that has become blurred in the multiple layers constructed as the encoder is constructed (for example, using skip connections) so that the information can be used in layers of the same dimension (layers corresponding to each other) in the multiple layers constructed as the decoder.

[0155] Although not shown in the drawings, as a modification example of the configuration of CNN, for example, after a convolution layer or the like, a batch normalization layer or an activation layer using a rectified linear unit may be incorporated.

[0156] When data is input into the learning model of such a machine learning model, data corresponding to the design of the machine learning model is output. For example, according to the tendency of training the machine learning model using the training data, output data having a high probability corresponding to the input data is output. When an original tomographic image is input into the learning model according to the present embodiment, a high-quality tomographic image for comprehensive observation in which the retina, vitreous body, and choroid are easily observed is output.

[0157] Note that in the case where learning is performed in a manner involving dividing a tomographic image into regions, the learning model outputs rectangular region images as high-quality tomographic images corresponding to the respective rectangular regions. In this case, first, the image quality improvement unit 322 divides the tomographic image as an input image into rectangular region image groups based on the image size when learning is performed, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improvement unit 322 arranges the respective images of a group of rectangular region images as high-quality tomographic images obtained using the learning model according to the same positional relationship as the positional relationship of the respective images of the rectangular region image group input into the learning model, and merges the rectangular region images. In this way, the image quality improvement unit 322 can generate a high-quality tomographic image corresponding to the input tomographic image.

[0158] (flow chart)

[0159] Next, refer to Fig.11 A series of image processing operations according to the present embodiment is described. Fig.11 is a flowchart of a series of image processing operations according to the present embodiment.

[0160] First, in step S1101 , the obtaining unit 310 obtains tomographic information obtained by imaging the subject's eye E. The obtaining unit 310 may obtain tomographic information of the subject's eye E using the imaging unit 20 , or may obtain tomographic information from the memory 340 or another device connected to the control unit 30 .

[0161] When obtaining tomographic information of the eye E using the imaging unit 20 , scanning of the eye E can be started after selecting an imaging mode and setting and adjusting various imaging parameters such as scanning mode, scanning range, focus, and fixation lamp position.

[0162] In step S1102 , the tomographic image generation unit 321 generates a tomographic image based on the acquired tomographic information of the subject's eye E. Note that in step S1101 , when the acquisition unit 310 acquires a tomographic image from the memory 340 or another device connected to the control unit 30 , step S1102 may be omitted.

[0163] In step S1103 , by using the learning model, the image quality improving unit 322 generates a high-quality tomographic image that appears to be an image subjected to different image processing for each region, based on the tomographic image generated in step S1102 or the tomographic image obtained in step S1101 .

[0164] Note that in the case where the learning model is learned in such a manner that the image is divided into regions, the image quality improving unit 322 first divides the tomographic image as the input image into rectangular region image groups based on the image size at the time of learning, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improving unit 322 arranges the respective images of a group of rectangular region images as the high-quality tomographic image obtained using the learning model according to the same positional relationship as the positional relationship of the respective images of the rectangular region image group input into the learning model, and merges the rectangular region images, thereby generating a final high-quality tomographic image.

[0165] In step S1104, the display control unit 350 causes the high-quality tomographic image generated in step S1103 to be displayed on the display unit 50. When the display processing by the display control unit 350 ends, a series of image processing operations ends.

[0166] According to such processing, a high-quality tomographic image that appears to have been subjected to different image processing in different regions using a learning model can be generated and displayed. In particular, in this embodiment, even for a diseased eye, etc., an image suitable for comprehensive observation in which the contrast of the vitreous body, choroid, and retina is enhanced can be generated and displayed.

[0167] As described above, the control unit 30 according to the present embodiment includes an acquisition unit 310 and an image quality improvement unit 322. The acquisition unit 310 acquires a first tomographic image (a tomographic image obtained by light interference) of the subject's eye E as the subject. The image quality improvement unit 322 generates a second tomographic image (a second medical image) from the first tomographic image (a first medical image) using a learning model, and the second tomographic image appears to be an image in which different image processing has been performed on different regions in the first tomographic image. In addition, in the present embodiment, the training data of the learning model includes tomographic images that have been subjected to grayscale conversion processing according to the region of the subject's eye E.

[0168] According to the aforementioned configuration, a high-quality tomographic image that appears to be an image in which different image processing has been performed on different regions using a learning model can be generated and displayed. In particular, in the present embodiment, even in the case where a satisfactory result cannot be obtained by segmentation of a tomographic image for a diseased eye or the like, a display image can be obtained through which the detailed internal structures of the retina, vitreous body, and choroid can be observed.

[0169] In addition, in the present embodiment, the image quality improvement unit 322 can generate a high-quality tomographic image in which the image quality of each region has been improved using a learning model. Therefore, the image quality improvement unit 322 can generate a second tomographic image from a first tomographic image, in which the image quality of different regions of the first region and the second region different from the first region in the first tomographic image has been improved using a learning model. Here, for example, the first region may be a region of the retina, and the second region may be a region of the vitreous body. In addition, the number of regions for which image quality improvement is performed is not limited to two, and may be three or more. In this case, for example, the third region that is different from the first region and the second region and for which image quality improvement is performed may be a region of the choroid. Note that the respective regions to be subjected to image quality improvement can be arbitrarily changed according to the desired configuration. Also from this point of view, the control unit 30 according to the present embodiment can generate an image that appears to be an image that has undergone appropriate image processing for the respective regions that are observation targets.

[0170] Although in the learning model according to the present embodiment, an image that has been subjected to grayscale conversion processing suitable for each region is used as output data of training data, the training data is not limited thereto. For example, a high-quality image obtained by performing averaging processing such as arithmetic averaging or maximum a posteriori processing (MAP estimation processing) on ​​each region of a tomographic image with respect to a source image group can be used as output data of training data. Here, the term "source image" refers to a tomographic image used as input data.

[0171] In the MAP estimation process, a likelihood function is obtained based on the probability density of each pixel value in a plurality of images, and the obtained likelihood function is used to estimate the true signal value (pixel value). The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values ​​close to the true signal value. In addition, since the estimated signal value is determined based on the probability density, randomly generated noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using a learning model, a high-quality image suitable for image diagnosis in which noise is reduced and which has a high contrast can be generated from an input image, and the learning model is learned using the high-quality image obtained by the MAP estimation process as training data. Note that for the method of generating pairs of input data and output data for training data, a method similar to the case where an average image is used as training data can be performed.

[0172] Furthermore, as output data of training data, a high-quality image obtained by applying a smoothing filter process using a mean filter or the like to a source image can be used. In this case, a high-quality image in which random noise is reduced can be generated from an input image by using a learning model. Note that, for a method for generating a pair of input data and output data of training data, a method similar to the case where an image subjected to grayscale conversion processing is used as training data can be performed.

[0173] Note that an image obtained from an image pickup device having the same image quality tendency as the image pickup unit 20 can be used as input data of training data. In addition, as output data of training data, a high-quality image obtained by a high-cost process such as a process using a successive approximation method can be used, or a high-quality image obtained by photographing an object corresponding to the input data using an image pickup device having a higher performance than the image pickup unit 20 can be used. In addition, a high-quality image obtained by performing a rule-based noise reduction process based on the structure of the object to be inspected, etc. can be used as output data. Here, the noise reduction process can include, for example, a process of replacing a high-luminance pixel of only one pixel, which is obviously noise appearing in a low-luminance area, with an average value of neighboring low-luminance pixel values. Therefore, an image photographed by a camera having a higher performance than the camera used to photograph the input image, or an image obtained by a camera process involving a greater number of steps than the camera process used to obtain the input image, can be used as training data for learning by the learning model.

[0174] In addition, the output data of the training data may be an image that has been subjected to the averaging process, MAP estimation process, etc. as described above, or may be an image obtained by performing different grayscale conversion processes on each region as an observation target for an image captured by an imaging device having higher performance than the imaging unit 20. Therefore, the output data of the training data may be a tomographic image generated using a combination of different grayscale conversion processes on each region as an observation target, and a tomographic image obtained by performing other processes related to image quality improvement or by imaging using a high-performance imaging device. In this case, a tomographic image more suitable for diagnosis can be generated and displayed.

[0175] In addition, although the original tomographic image is used as input data in the present embodiment, the input data is not limited thereto. For example, a tomographic image that has been gray-scale converted so that the retina portion is easy to observe or a tomographic image that has been gray-scale converted so that the vitreous body portion and the choroid portion are easy to observe may be used as input data. In this case, the image quality improvement unit 322 may input the tomographic image that has been gray-scale converted so that the retina portion or the vitreous body portion and the choroid portion are easy to observe corresponding to the input data of the training data into the learning model, thereby generating a high-quality tomographic image.

[0176] In addition, data with a high dynamic range adjusted to data that can easily perform appropriate grayscale conversion on each region can be used as output data. In this case, the image quality improvement unit 322 can generate a high-quality tomographic image by performing appropriate grayscale conversion on the data with a high dynamic range obtained using the learning model.

[0177] Note that although it has been described that the image quality improvement unit 322 generates a high-quality image to which appropriate grayscale conversion is performed for display on the display unit 50 by using a learning model, the image quality improvement processing of the image quality improvement unit 322 is not limited to this. It is sufficient that the image quality improvement unit 322 can generate an image with image quality that is more suitable for image diagnosis.

[0178] In a tomographic image obtained using a learning model, there may also be a case where, according to a learning trend, tissues such as blood vessels that do not actually exist are visualized, or tissues that should exist are not visualized. Therefore, when displaying a high-quality tomographic image obtained using a learning model, the display control unit 350 may also display information to the effect that the image is a tomographic image obtained using a learning model. In this case, it is possible to suppress the operator from making a misdiagnosis. Note that the display form may be any form as long as the form makes it understandable that the image is an image obtained using a learning model.

[0179] (Variant 1)

[0180] In Embodiment 1, the case has been described where an image of a local region (rectangular region) of a tomographic image that has been subjected to grayscale conversion processing so as to be fully observed is used as output data of training data. In contrast, in Modification 1, a tomographic image in which each region as an observation target is different is used as output data of training data. FIG. 12A to FIG. 12C Describe the training data in this variant. Note that since the components and processing operations according to this variant except for the training data of the machine learning model are the same as those in Example 1, these components and processing operations are represented by the same figure marks as in Example 1 and their description is omitted below.

[0181] Fig. 12A An example of an original tomographic image 1210 related to input data of training data is shown. Fig. 12A , a rectangular area image 1212 of the vitreous body region, a rectangular area image 1211 of the retinal region, and a rectangular area image 1213 of the choroid region are shown.

[0182] Fig. 12B A tomographic image 1220 is shown which is obtained by performing grayscale conversion processing to ensure contrast in the region of the retina portion of the original tomographic image 1210. Fig. 12B , a rectangular area image 1221 having a corresponding positional relationship with the rectangular area image 1211 of the retinal region is shown.

[0183] Fig. 12C A tomographic image 1230 is shown which is obtained by performing grayscale conversion processing to ensure contrast in the vitreous body and choroid regions of the original tomographic image 1210. Fig. 12C , a rectangular area image 1232 having a corresponding positional relationship with the rectangular area image 1212 of the vitreous body region, and a rectangular area image 1233 having a corresponding positional relationship with the rectangular area image 1213 of the choroid region are shown.

[0184] In the present modification, a pair of training data is created by taking a rectangular region image 1211 of a region of the retina portion in the original tomographic image 1210 as input data, and taking a rectangular region image 1211 of a region of the retina portion in the tomographic image 1220 as output data. Similarly, a pair of training data is created by taking a rectangular region image 1212 of a region of the vitreous portion in the original tomographic image 1210 as input data, and taking a rectangular region image 1232 of a region of the vitreous portion in the tomographic image 1230 as output data. Furthermore, a pair of training data is created by taking a rectangular region image 1213 of a region of the choroid portion in the original tomographic image 1210 as input data, and taking a rectangular region image 1233 of a region of the choroid portion in the tomographic image 1230 as output data.

[0185] In this case as well, a tomographic image obtained by performing appropriate grayscale conversion processing on each region as an observation target can be used as output data of training data. Therefore, similar to Embodiment 1, the image quality improvement unit 322 can use a learning model (which is learned using such training data) to generate a high-quality tomographic image that appears to be an image in which different image processing has been performed on each region as an observation target.

[0186] (Variant 2)

[0187] In Embodiment 1, training data of a machine learning model is used, in which a tomographic image obtained by performing image quality improvement processing such as grayscale conversion processing on an original tomographic image without considering the imaging mode is adopted as output data. In this regard, in the OCT apparatus, a strong or weak tendency for signal brightness in a tomographic image differs depending on the imaging mode. Therefore, in Modification 2, for each region as an observation target, a tomographic image obtained in an imaging mode in which there is a tendency for signal brightness to be high for the relevant region is used as output data of the training data.

[0188] In the following, reference is made to FIG. 13A to FIG. 15C Describe the training data according to the present modification. Note that since the components and processing operations other than the training data of the machine learning model according to the present modification are the same as those of Example 1, these components and processing operations are denoted by the same reference numerals as those of Example 1 and their description is omitted below. First, the imaging methods in the vitreous mode and the choroidal mode will be described as the imaging methods of the respective imaging modes of the OCT apparatus 1.

[0189] (Photography method in glass body mode)

[0190] Reference FIG. 13A to FIG. 13C, the imaging method in the vitreous body mode of the OCT apparatus 1 will be described. In the vitreous body mode, as Fig.13A As shown, the reference mirror 221 is moved so that a position Z1 in the depth direction (Z-axis direction) where the optical path lengths of the reference light and the measuring light match is located on the shallow side (vitreous body side) in the depth direction of the imaging range C10, and imaging is performed.

[0191] In this case, if Fig. 13B As shown, relative to position Z1, a normal image is obtained in imaging range C10 in the positive direction of the Z direction, and a virtual image is obtained in imaging range C11 in the negative direction. Imaging in the vitreous mode of the OCT apparatus is usually performed by obtaining a normal image in imaging range C10 as a tomographic image. Fig. 13C , which is an example of a tomographic image obtained in the vitreous body mode. Note that a virtual image on the imaging range C11 side can also be obtained as the tomographic image C12. In the case where a virtual image on the imaging range C11 side is obtained as the tomographic image C12, the image can be vertically inverted and displayed.

[0192] In the OCT apparatus, the closer the region is to the position in the depth direction where the optical path lengths of the reference light and the measurement light match, the higher the signal brightness obtained for the relevant region becomes. Therefore, in the tomographic image C12 captured in the vitreous body mode, the signal brightness on the side closer to the position Z1, that is, the vitreous body side, will be higher.

[0193] (Imaging method in choroidal mode)

[0194] Next, refer to FIG. 14A to FIG. 14C , the imaging method in the choroidal mode of the OCT apparatus will be described. In the choroidal mode, as Fig.14A As shown, the reference mirror 221 is moved so that the position Z2 in the depth direction where the optical path lengths of the reference light and the measuring light match is located on the deep side (choroid side) in the depth direction of the imaging range, and imaging is performed.

[0195] In this case, if Fig. 14B As shown, relative to position Z2, a normal image is obtained in imaging range C20 in the negative direction of the Z direction, and a virtual image is obtained in imaging range C21 in the positive direction. Imaging in the choroidal mode of the OCT apparatus is usually performed by obtaining a virtual image on the imaging range C21 side as a tomographic image. Fig. 14C, which is an example of a tomographic image obtained in the choroidal mode. Note that a normal image on the imaging range C20 side can also be obtained as the tomographic image C22. Furthermore, in the case where a virtual image on the imaging range C21 side is obtained as the tomographic image C22, the image can be vertically inverted and displayed.

[0196] As described above, in the OCT apparatus, the closer the region is to the position in the depth direction where the optical path lengths of the reference light and the measurement light match, the higher the signal brightness obtained for the relevant region becomes. Therefore, in the tomographic image C22 captured in the choroidal mode, the signal brightness on the side closer to the position Z2, that is, the choroid side, will be higher.

[0197] Taking this characteristic of the OCT device into consideration, in the present modification, according to the region as the observation target, a tomographic image obtained in an imaging mode having a tendency to make the signal brightness of the relevant region in particular higher is used as output data of the training data of the machine learning model. More specifically, in the OCT device, in a tomographic image imaged in a vitreous mode, the signal brightness of the vitreous side will be higher, and in a tomographic image imaged in a choroidal mode, the signal brightness of the choroidal side will be higher. Therefore, the same part of the same eye to be inspected is imaged in the choroidal mode and the vitreous mode, and for each local area image (rectangular area image) of the input data, a tomographic image in which the signal brightness of the corresponding local area is higher is used as output data. In other words, in the present modification, the training data of the learning model includes a medical image obtained by imaging the subject and obtained in an imaging mode corresponding to any different region in the relevant medical image.

[0198] Fig.15A An example of an original tomographic image 1510 related to input data of training data captured in the vitreous body mode is shown. Fig.15A , a rectangular area image 1511 of the vitreous body region and a rectangular area image 1512 of the choroid region are shown.

[0199] Fig. 15B A tomographic image 1520 is shown which is obtained by performing grayscale conversion processing on a tomographic image obtained by imaging the same part of the same subject's eye in the vitreous body mode so as to ensure contrast in the regions of the retina, vitreous body, and choroid. Fig. 15B , a rectangular area image 1521 having a corresponding positional relationship with the rectangular area image 1511 of the vitreous body region is shown.

[0200] Fig. 15CA tomographic image 1530 is shown which is obtained by performing grayscale conversion processing on a tomographic image obtained by imaging the same part of the same subject's eye in the choroid mode so as to ensure contrast in the retinal, vitreous, and choroidal regions. Fig. 15C , a rectangular area image 1532 having a corresponding positional relationship with the rectangular area image 1512 of the choroid region is shown.

[0201] In the present modification, a pair of training data is created by taking a rectangular region image 1511 of the region of the vitreous body portion in the original tomographic image 1510 as input data, and taking a rectangular region image 1521 of the region of the vitreous body portion in the tomographic image 1520 as output data. Similarly, a pair of training data is created by taking a rectangular region image 1512 of the region of the choroid portion in the original tomographic image 1510 as input data, and taking a rectangular region image 1532 of the region of the choroid portion in the tomographic image 1530 as output data. Note that in the present modification, since the tomographic image 1530 captured in the choroid mode is vertically inverted with respect to the original tomographic image 1510 related to the input data, the rectangular region image obtained by vertically inverting the rectangular region image 1532 is used as output data of the training data.

[0202] In this case, for a tomographic image obtained in an imaging mode according to an area as an observation target, which is an imaging mode having a tendency to make the signal brightness of a particularly relevant area higher, a tomographic image obtained by performing a grayscale conversion process on the relevant tomographic image according to the area can be used as output data of the training data. In other words, the training data of the learning model may include a medical image obtained by imaging the subject and a medical image obtained by performing a grayscale conversion process corresponding to any different area in the relevant medical image on a medical image obtained in an imaging mode corresponding to any different area in the relevant medical image. By using a learning model learned with the aid of such training data, the image quality improving unit 322 can generate a tomographic image that appears to be an image whose image quality has been further improved for each area as an observation target.

[0203] Note that the input data of the training data is not limited to the original tomographic image captured in the vitreous mode, and may be the original tomographic image captured in the choroidal mode. In this case, since the tomographic image captured in the vitreous mode is vertically inverted with respect to the original tomographic image related to the input data, an image obtained by vertically inverting the rectangular area image related to the tomographic image captured in the vitreous mode is used as the output data of the training data.

[0204] Furthermore, the grayscale conversion processing applied to the tomographic images captured in various imaging modes is not limited to the grayscale conversion processing for ensuring the contrast of the regions of the retina, vitreous body, and choroidal parts so that comprehensive observation can be performed. For example, similar to Modification 1, a tomographic image obtained by grayscale conversion of a tomographic image captured in the vitreous body mode to ensure the contrast of the region of the vitreous body part can be used as output data of the training data. Similarly, a tomographic image obtained by grayscale conversion of a tomographic image captured in the choroidal mode to ensure the contrast of the region of the choroidal part can be used as output data of the training data.

[0205] Note that output data based on a tomographic image captured in a vitreous body mode or output data based on a tomographic image captured in a choroidal mode can be used as output data of training data related to an area of ​​the retinal part. Furthermore, the imaging mode is not limited to the vitreous body mode and the choroidal mode, and can be arbitrarily set according to a desired configuration. Also in this case, based on a strong or weak tendency for signal brightness in a tomographic image according to the imaging mode, for each area as an observation target, a tomographic image having a tendency to make the signal brightness of the relevant area higher can be used as output data of training data.

[0206] Furthermore, also in Modifications 1 and 2, similar to Embodiment 1, the input data of the training data is not limited to the original tomographic image, and may be a tomographic image subjected to arbitrary grayscale conversion. Furthermore, the output data of the training data is not limited to the tomographic image subjected to grayscale conversion, and may be a tomographic image obtained by making adjustments relative to the original tomographic image so that grayscale conversion can be easily performed.

[0207] (Variant 3)

[0208] In Embodiment 1, the image quality improvement unit 322 uses a learning model to generate a high-quality image that appears to be an image in which different image processing has been performed on each region of the target image. In contrast, according to Modification 3, first, for a tomographic image as input data, the image quality improvement unit 322 uses a first learning model to generate a label image in which each pixel has been labeled (annotated) by region. Thereafter, for the generated label image, the image quality improvement unit 322 uses a second learning model different from the first learning model to generate a high-quality image that appears to be an image in which image processing has been performed on it according to each region. In other words, the image quality improvement unit 322 generates a label image from a tomographic image (first medical image) as input data using a learning model different from the learning model used to generate the high-quality image (second medical image), in which different label values ​​are assigned to different regions. In addition, the image quality improvement unit 322 generates a high-quality image from the label image by using the learning model used to generate the high-quality image (second medical image).

[0209] In this modification, for the first learning model, learning is performed using training data in which a tomographic image is used as input data, and a label image in which each pixel of the tomographic image has been labeled as a region is used as output data. Note that for the label image, an image properly processed by conventional segmentation processing can be used, or a manually labeled label image can be used. The label can be, for example, a vitreous label, a retinal label, a choroid label, etc. Note that the label can be displayed using a character string, or can be a pre-set numerical value corresponding to each region. In addition, the label is not limited to the above example, and any region can be indicated according to the desired construction.

[0210] Furthermore, for the second learning model, learning is performed using training data in which a label image is used as input data, and a tomographic image obtained by performing image quality improvement processing corresponding to the label of each pixel on the relevant label image is used as output data. Note that, for the image quality improvement processing corresponding to the label of each pixel, as described above, the image quality improvement processing may include grayscale conversion processing according to the area as an observation target, etc.

[0211] In this case, similar to Embodiment 1, the image quality improvement unit 322 can generate a high-quality tomographic image that appears to be an image in which different image processing has been performed on each region as an observation target using the first and second learning models. In addition, the learning model outputs output data having a high probability of corresponding to the input data according to the learning trend. In this regard, when the learning model uses a group of images having similar image quality trends to each other as training data for learning, an image that has undergone more effective image quality improvement can be output relative to the image having the similar trend in question. Therefore, as in the present variant, by using a learning model that utilizes training data in which each region is labeled, it can be expected that an image that has undergone more effective image quality improvement will be generated.

[0212] Note that, also with respect to the training data related to this variant, similar to Example 1, the entire image can be used, or a rectangular area image (partial image) can be used. In addition, depending on the desired configuration, the input data and the output data can be images after any grayscale conversion or can be images before grayscale conversion.

[0213] (Variant 4)

[0214] In Embodiment 1, the following case is described: the image quality improvement unit 322 integrates the partial images of the tomographic image obtained using the learning model, thereby generating a final high-quality tomographic image. In particular, in the example described in Embodiment 1, the partial image obtained using the learning model is an image that appears to be an image in which different grayscale conversion processes have been performed on each region as an observation target according to the learning trend. Therefore, if the partial images are simply integrated, in some cases, the luminance distribution will be significantly different between the position where different regions touch (connected portion) and the position of the region adjacent to the position (for example, the region of the vitreous body or the region of the retinal body), and the image edge will be conspicuous.

[0215] Therefore, in Modification 4, when integrating the local images obtained using the learning model, the image quality improvement unit 322 modifies the pixel values ​​of the connected portion of the area as the observation target based on the pixel values ​​of the surrounding pixels, thereby making the image edge inconspicuous. In this way, an image suitable for diagnosis can be generated in which the sense of incongruity caused by the image edge is reduced.

[0216] In this case, the image quality improvement unit 322 may modify the luminance value by performing any known blending process for the connected portion of the area being the observation target. Note that the image quality improvement unit 322 may also perform blending process for a position adjacent to the connected portion of the area being the observation target. Furthermore, the process of making the image edge inconspicuous is not limited to blending process, and may be any other process.

[0217] (Example 2)

[0218] In Embodiment 1, image quality improvement processing using a learning model is uniformly performed for generated / acquired tomographic images. In contrast, in the OCT apparatus according to Embodiment 2, image processing to be applied to tomographic images is selected according to an instruction of an operator.

[0219] In the following, reference is made to Figures 16 to 18C , the OCT device according to the present embodiment is described. Note that the configuration except for the control unit according to the present embodiment is the same as that of the OCT device 1 according to Embodiment 1, and therefore the same components are denoted by the same reference numerals as in Embodiment 1, and description thereof is omitted hereinafter. Hereinafter, the OCT device according to the present embodiment will be described centering on the differences from the OCT device according to Embodiment 1.

[0220] Fig.16 1 shows a schematic configuration example of the control unit 1600 according to the present embodiment. Note that in the control unit 1600, the configuration other than the image quality improvement unit 1622 and the selection unit 1623 of the image processing unit 1620 is the same as that of the control unit 30 according to Embodiment 1, and therefore the same components are denoted by the same reference numerals as in Embodiment 1, and description thereof is omitted hereinafter.

[0221] The image processing unit 1620 includes an image quality improving unit 1622 and a selecting unit 1623 in addition to the tomographic image generating unit 321. The selecting unit 1623 selects image processing to be applied to the tomographic image according to an instruction input through the input unit 40 from the operator.

[0222] The image quality improving unit 1622 applies the image processing selected by the selecting unit 1623 to the tomographic image generated by the tomographic image generating unit 321 or the tomographic image obtained by the obtaining unit 310 , thereby generating a high-quality tomographic image.

[0223] Next, refer to Fig.17 A series of image processing operations according to the present embodiment is described. Fig.17 0 is a flowchart of a series of image processing operations according to the present embodiment. Note that steps S1701 and S1702 are the same as steps S1101 and S1102 according to Embodiment 1, and thus descriptions of these steps will be omitted here.

[0224] When the tomographic image generation unit 321 generates the original tomographic image in step S1702, the process proceeds to step S1703. In step S1703, the acquisition unit 310 acquires instructions from the operator regarding the selection of a region of interest in the tomographic image or the processing to be performed on the tomographic image. Note that at this time, the display control unit 350 can cause the processing options to be displayed on the display unit 50 to present the options to the operator.

[0225] In step S1704, the selection unit 1623 selects the image processing (image quality improvement processing) applied to the tomographic image according to the instruction from the operator obtained in step S1703. In the present embodiment, according to the instruction from the operator, the selection unit 1623 selects the image quality improvement processing for the retina portion, the image quality improvement processing for the vitreous / choroid portion, or the image quality improvement processing for the entire image.

[0226] If the image quality improvement process for the retina is selected in step S1704, the process proceeds to step S1705. In step S1705, the image quality improvement unit 1622 performs grayscale conversion processing on the original tomographic image so that the retina region becomes easy to observe, thereby generating a high-quality tomographic image.

[0227] If the image quality improvement process for the vitreous body / choroid is selected in step S1704, the process proceeds to step S1706. In step S1706, the image quality improvement unit 1622 performs grayscale conversion processing on the original tomographic image as described above so that the regions of the vitreous body and the choroid become easy to observe, thereby generating a high-quality tomographic image.

[0228] If the image quality improvement process for the entire image is selected in step S1704, the process proceeds to step S1707. In step S1707, based on the original tomographic image, the image quality improvement unit 1622 generates a high-quality tomographic image in which the retinal part, the vitreous body part, and the choroid part are easy to observe using the learning model. Note that the learning model according to the present embodiment is similar to the learning model according to Embodiment 1, so the description of the learning model and the training data is omitted here.

[0229] In step S1708, the display control unit 350 causes the high-quality tomographic image generated in step S1705, step S1706, or step S1707 to be displayed on the display unit 50. When the display processing by the display control unit 350 ends, a series of image processing operations ends.

[0230] Now refer to 18A to 18C The operation method according to the present embodiment is described. 18A to 18CAn example of a display screen including options regarding a region to be noticed and a tomographic image subjected to image quality improvement processing according to the selected region is shown.

[0231] Fig.18A A display screen 1800 is shown in which the retinal region is selected as the region of interest. On the display screen 1800, an option 1801 and a tomographic image 1802 on which a grayscale conversion process has been performed so that the retinal region becomes easy to observe are displayed.

[0232] When the operator wishes to focus on the area of ​​the retina, the operator selects "retina" from among the three options of "retina", "vitreous body / choroid" and "whole" in the option 1801 through the input unit 40. The selection unit 1623 selects the image quality improvement processing for the area of ​​the retina according to the instruction from the operator, and the image quality improvement unit 1622 applies the selected image quality improvement processing to the tomographic image to generate a tomographic image 1802 in which the retina is easily observed. The display control unit 350 displays the generated tomographic image 1802 in which the retina is easily observed on the display screen 1800.

[0233] Fig.18B A display screen 1810 is shown in which the vitreous body and choroid regions are selected as the regions of interest. Display screen 1810 displays an option 1811 and a tomographic image 1812 that has been subjected to grayscale conversion processing so that the vitreous body and choroid regions are easily observed.

[0234] In the case where the operator wishes to focus on the region of the vitreous body and the choroid, the operator selects "vitreous body / choroid" from among the three options of "retina", "vitreous body / choroid" and "whole" in the option 1801 through the input unit 40. The selection unit 1623 selects the image quality improvement processing for the region of the vitreous body and the choroid according to the instruction from the operator, and the image quality improvement unit 1622 applies the selected image quality improvement processing to the tomographic image to generate a tomographic image 1812 with high image quality in which the vitreous body and the choroid are easily observed. The display control unit 350 displays the generated tomographic image 1812 in which the vitreous body and the choroid are easily observed on the display screen 1810.

[0235] Fig.18C A display screen 1820 is shown in which a region of the entire image is selected as a region of interest. On the display screen 1820, options 1821 and a tomographic image 1822 that appears to be an image subjected to gradation conversion processing so that the region of the entire image becomes easy to observe are displayed.

[0236] In the case where the operator wishes to focus on the area of ​​the entire image, the operator selects "whole" from among the three options of "retina", "vitreous body / choroid" and "whole" in the option 1821 through the input unit 40. The selection unit 1623 selects the image quality improvement processing for the entire image according to the instruction from the operator, and the image quality improvement unit 1622 applies the selected image quality improvement processing to the tomographic image to generate a high-quality tomographic image. In this case, the image quality improvement unit 1622 generates a high-quality tomographic image in which the entire image is easy to observe using the learned model. The display control unit 350 displays the generated tomographic image 1822 in which the area of ​​the entire image is easy to observe on the display screen 1820.

[0237] As described above, the control unit 1600 according to the present embodiment includes the selection unit 1623, which selects, according to an instruction from the operator, image processing to be applied to the first tomographic image obtained by the obtaining unit 310. Based on the image processing selected by the selection unit 1623, the image quality improving unit 1622 generates a third tomographic image (third medical image) by performing grayscale conversion processing on the first tomographic image without using a learning model, or generates a second tomographic image from the first tomographic image using a learning model.

[0238] According to the present configuration, the control unit 1600 can generate a tomographic image to which different image processing has been performed according to the region that the operator wishes to focus on. In particular, as described above, when image quality improvement processing using a learning model is performed, in some cases, tissue that does not actually exist is visualized, or tissue that originally existed is not visualized. Therefore, misdiagnosis can be prevented by observing and comparing tomographic images that have been subjected to different image processing.

[0239] Furthermore, it is not necessary that the grayscale conversion processing performed so that the retinal area becomes easy to observe or the grayscale conversion processing performed so that the vitreous body and choroidal areas become easy to observe is segmentation processing. Therefore, even for a diseased eye, appropriate image quality improvement processing can be expected.

[0240] In the present embodiment, an example has been described in which, after obtaining the operator's instruction regarding the desired region of interest in step S1703, image processing is performed according to the instruction. However, the order of obtaining the instruction from the operator and performing the image processing is not limited to this. The original tomographic image may be image processed in advance according to each option by the image quality improvement unit 1622, thereby generating each high-quality tomographic image in advance, and only the high-quality tomographic image to be displayed may be switched according to the operator's instruction. In this case, the selection unit 1623 may function as a selection unit for selecting the high-quality tomographic image to be displayed.

[0241] Furthermore, after applying the preset image processing (default image processing) to the original tomographic image to generate a high-quality tomographic image and displaying the high-quality tomographic image, an instruction from the operator may be obtained. In this case, if an instruction to perform image processing other than the default image processing is obtained from the operator, a new high-quality image generated by performing image processing according to the instruction may be displayed.

[0242] Note that although an example has been described in which the same image processing is performed on the regions of the vitreous body portion and the choroid portion, separate image processing may be performed on the region of the vitreous body portion and the region of the choroid portion.

[0243] Furthermore, the image processing is not limited to the image quality improvement processing for the region of the retina, the image quality improvement processing for the region of the vitreous body and the choroidal body, and the image quality improvement processing using the learning model. For example, as described above, a grayscale conversion processing that makes the regions of the retina, the vitreous body, and the choroidal body easy to observe based on the premise of performing the segmentation processing can be included in the image processing options. In this case, a high-quality tomographic image generated by the image processing based on the premise of performing the segmentation processing and a high-quality tomographic image generated by the image processing using the learning model and the like can be observed and compared. Therefore, the operator can easily determine the false detection caused by the segmentation processing and the authenticity of the tissue in the tomographic image generated using the learning model.

[0244] (Example 3)

[0245] In Embodiment 1, an image subjected to image quality improvement processing using a learning model is displayed. In contrast, in the case of the OCT apparatus according to Embodiment 3, image analysis in which different analysis conditions are applied to each of a plurality of regions different from each other in the generated high-quality tomographic image is performed and the analysis result is displayed.

[0246] In the following, reference is made to Fig.19 and Fig. 20 , the OCT device according to the present embodiment is described. Note that the configuration except for the control unit according to the present embodiment is the same as that of the OCT device 1 according to Embodiment 1, and therefore the same components are denoted by the same reference numerals as in Embodiment 1, and description thereof is omitted hereinafter. Hereinafter, the OCT device according to the present embodiment will be described centering on the differences from the OCT device according to Embodiment 1.

[0247] Fig.19A schematic configuration example of a control unit 1900 according to the present embodiment is shown. Note that in the control unit 1900, the configuration other than the analysis unit 1924 of the image processing unit 1920 is the same as that of the control unit 30 according to Embodiment 1, and therefore the same components are denoted by the same reference numerals as in Embodiment 1, and description thereof is omitted hereinafter.

[0248] In addition to the tomographic image generation unit 321 and the image quality improvement unit 322, the image processing unit 1920 also includes an analysis unit 1924. The analysis unit 1924 performs image analysis on the high-quality tomographic image generated by the image quality improvement unit 322 based on the analysis conditions set for each region. Here, as the analysis conditions set for each region, for example, layer extraction or blood vessel extraction is set for the region of the retinal part and the region of the choroidal part, and detection of vitreous or vitreous detachment is set for the region of the vitreous body part. Note that the analysis conditions may be set in advance or may be appropriately set by the operator.

[0249] In the case where layer extraction is set as the analysis condition, the analysis unit 1924 performs layer extraction for the region where the analysis condition in question is set, and can perform layer thickness value measurement, etc. for the extracted layer. In addition, in the case where blood vessel extraction is set as the analysis condition, the analysis unit 1924 performs blood vessel extraction for the region where the analysis condition in question is set, and can perform blood vessel density measurement, etc. for the extracted blood vessels. In addition, in the case where the detection of vitreous or vitreous detachment is set as the analysis condition, the analysis unit 1924 performs the detection of vitreous or vitreous detachment for the region where the analysis condition in question is set. Thereafter, the analysis unit 1924 can quantify the detected vitreous or vitreous detachment, and determine the thickness, width, area or volume, etc. of the vitreous or vitreous detachment.

[0250] Please note that the analysis conditions are not limited to the above conditions and can be arbitrarily set according to the desired structure. For example, the detection of the fiber structure of the vitreous body can be set for the region of the vitreous body. In this case, the analysis unit 1924 can quantify the fiber structure of the detected vitreous body and determine the thickness, width, area or volume of the fiber structure. In addition, the analysis processing according to the analysis conditions is also not limited to the above processing and can be arbitrarily set according to the desired structure.

[0251] The display control unit 350 can cause the result of the image analysis performed by the analysis unit 1924 to be displayed on the display unit 50 together with the high-quality tomographic image or separately from the high-quality tomographic image.

[0252] Next, refer to Fig. 20 A series of image processing operations according to the present embodiment is described. Fig. 200 is a flowchart of a series of image processing operations according to the present embodiment. Note that since steps S2001 to S2003 are the same as steps S1101 to S1103 according to Embodiment 1, description of these steps will be omitted hereinafter.

[0253] Similar to step S1103, when the image quality improvement unit 322 generates a high-quality tomographic image in step S2003, the process proceeds to step S2004. In step S2004, the analysis unit 1924 performs segmentation processing on the generated high-quality tomographic image, thereby detecting a plurality of different regions in the tomographic image. As the plurality of regions, the analysis unit 1924 can detect, for example, a region of the vitreous body, a region of the retinal part, and a region of the choroidal part. Note that any known method can be used as a method of the segmentation processing, and, for example, the segmentation processing can be a rule-based segmentation processing. Here, the term "rule-based processing" refers to processing that utilizes, for example, known regularities (such as regularities of the retinal shape).

[0254] Thereafter, the analysis unit 1924 performs image analysis for each detected region based on the analysis conditions set for each detected region. For example, according to the analysis conditions, the analysis unit 1924 performs layer extraction or blood vessel extraction on the region for which the analysis conditions in question have been set, and calculates the layer thickness or blood vessel density. Note that the layer extraction or blood vessel extraction can be performed by any known segmentation processing, etc. In addition, according to the analysis conditions, the analysis unit 1924 can detect the vitreous body, vitreous body shedding, or the fibrous structure of the vitreous body, and can quantify these. Note that when detecting the vitreous body, vitreous body shedding, or the fibrous structure of the vitreous body, the analysis unit 1924 can additionally perform contrast enhancement, binarization, morphological processing, boundary tracking processing, etc.

[0255] In step S2005, the display control unit 350 causes the various analysis results (e.g., layer thickness, blood vessel density, area of ​​vitreous body) obtained by the analysis of the analysis unit 1924 to be displayed on the display unit 50 together with the high-quality tomographic image generated by the image quality improvement unit 322. Note that the form in which the analysis results are displayed may be any form according to a desired configuration. For example, the display control unit 350 may display the analysis results of the respective regions in association with the respective regions of the high-quality tomographic image. In addition, the display control unit 350 may cause the analysis results to be displayed separately from the high-quality tomographic image on the display unit 50. When the display processing of the display control unit 350 ends, a series of image processing operations ends.

[0256] Therefore, the control unit 1900 according to the present embodiment includes the analysis unit 1924 that performs image analysis by applying different analysis conditions to regions different from each other in the high-quality tomographic image (second tomographic image) generated by the image quality improving unit 322. The display control unit 350 causes the analysis result for each of the plurality of regions different from each other in the high-quality tomographic image obtained by the analysis unit 1924 to be displayed on the display unit 50.

[0257] According to the present configuration, in order to perform image analysis on the high-quality tomographic image generated by the image quality improvement unit 322, the analysis unit 1924 can more appropriately detect features within the image, etc., and perform image analysis with higher accuracy. In addition, by performing image analysis according to the analysis conditions set for each region of the high-quality tomographic image that appears to be an image for which appropriate image processing has been performed on each region, the analysis unit 1924 can output appropriate analysis results for each region. Therefore, the operator can quickly obtain appropriate analysis results for the eye to be inspected.

[0258] In this embodiment, the analysis unit 1924 automatically performs image analysis based on the analysis conditions for each region of the high-quality tomographic image. Alternatively, the analysis unit 1924 may start image processing for the high-quality tomographic image based on an instruction from the operator.

[0259] The analysis unit 1924 according to the present embodiment can also be applied to the control unit 1600 according to Embodiment 2. In this case, the analysis unit 1924 can perform the above-mentioned image analysis for the tomographic images generated in steps S1705 to S1707, or perform image processing only for the region desired to be observed selected in step S1704. Note that in the case where segmentation processing is performed when image quality improvement is performed, the analysis unit 1924 can perform the above-mentioned image analysis for the high-quality tomographic images using the result of the segmentation processing.

[0260] In addition, in the present embodiment, the analysis unit 1924 performs segmentation processing on the high-quality tomographic image generated by the image quality improvement unit 322, and detects regions that are different from each other. In this regard, for example, in the case where the analysis unit 1924 is applied to the control unit according to Modification 3 of Embodiment 1, based on the label image obtained using the first learning model, the analysis unit 1924 can determine a plurality of different regions in the high-quality tomographic image.

[0261] (Variant 5)

[0262] In addition, for the tomographic image, the image processing unit 320, 1620 or 1920 can use a learning model for segmentation to generate a label image and perform segmentation processing. Here, the term "label image" refers to a label image in which each pixel is given a label of the region for the tomographic image as described above. Specifically, the term "label image" refers to an image that classifies any area among a group of areas visualized in the image according to a group of identifiable pixel values ​​(hereinafter referred to as "label values"). Here, the identified arbitrary areas include a region of interest (ROI) and a volume of interest (VOI), etc.

[0263] When a coordinate group of pixels having an arbitrary label value is identified from an image, a coordinate group that visualizes pixels of a corresponding region such as a retinal layer in the image can be identified. Specifically, for example, in a case where a label value indicating a ganglion cell layer forming the retina is "1", a coordinate group having a pixel value of "1" is identified among pixel groups in the image, and a pixel group corresponding to the coordinate group is extracted from the image. In this way, the region of the ganglion cell layer in the image can be identified.

[0264] Note that the segmentation process may include a process of reducing or enlarging the label image. In this case, a nearest neighbor method that does not mistakenly generate undefined label values ​​or label values ​​that should not exist at the corresponding coordinates may be used as an image interpolation process method for reducing or enlarging the label image.

[0265] The segmentation process will now be described in detail. The term "segmentation process" refers to a process of identifying a region called ROI or VOI such as an organ or a lesion visualized in an image so as to use such region for image diagnosis or image analysis. For example, according to the segmentation process, a region of a group of layer groups constituting the retina can be identified from an image obtained by OCT imaging for which the posterior segment of the eyeball is taken as an imaging target. Note that if the region to be identified is not visualized in the relevant image, the number of identified regions is 0. In addition, if a group of multiple regions to be identified are visualized in the image, the number of identified regions may be multiple, or may be a single region surrounding a group of regions so as to include the group of regions.

[0266] The identified area groups are output as information that can be used in other processing. Specifically, for example, the coordinate groups of the pixel groups constituting the identified area groups can be output as numerical data groups, respectively. In addition, for example, the coordinate groups indicating the rectangular areas, elliptical areas, parallelepiped areas, ellipsoidal areas, etc. including the identified area groups can also be output as numerical data groups. In addition, for example, the coordinate groups indicating the straight lines, curves, planes, curved surfaces, etc. as the boundaries of the identified area groups can be output as numerical data groups. In addition, for example, a label image indicating the identified area groups can also be output.

[0267] In this case, for example, a convolutional neural network (CNN) can be used as a machine learning model for segmentation. Note that, for example, according to this variation, Fig.10 The CNN (U-Net type machine learning model) shown in or a model obtained by combining CNN and LSTM (Long Short-Term Memory) can be used as a machine learning model. In addition, FCN (Fully Convolutional Network) or SegNet, etc. can also be used as a machine learning model. In addition, a machine learning model for object recognition in units of regions, etc. can be used according to the desired construction. As a machine learning model for object recognition in units of regions, for example, RCNN (Regional CNN), Fast-RCNN, or Faster-RCNN can be used. In addition, YOLO (Look Only Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model for object recognition in units of regions. Please note that the machine learning model described here as an example can be applied to the first learning model described in Variant 3.

[0268] In addition, for the training data of the machine learning model for segmentation, a tomographic image is used as input data, and a label image obtained by giving a label of a region to each pixel in the tomographic image is used as output data. As a label image, for example, a label image can be used in which labels such as the internal limiting membrane (ILM), nerve fiber layer (NFL), ganglion cell layer (GCL), photoreceptor inner segment-outer segment junction (ISOS), retinal pigment epithelium (RPE), Bruch's membrane (BM) and choroid are given to each region. Note that as other regions, for example, a label image can be used in which labels such as the vitreous body, sclera, outer plexiform layer (OPL), outer nuclear layer (ONL), inner plexiform layer (IPL), inner nuclear layer (INL), cornea, anterior chamber, iris and lens are given to each region. Note that for the label image described here as an example, the label image can also be used as output data of the training data related to the first learning model mentioned in Modification 3.

[0269] In addition, the input data of the machine learning model used for segmentation is not limited to the tomographic image. The input data may be an anterior segment image, an SLO fundus image, a frontal fundus image obtained using a fundus camera or the like, or an en-face image or an OCTA frontal image described later. In this case, for the training data, various images may be used as input data, and a labeled image in which the names of regions and the like have been labeled for each pixel of the various images may be used as output data. For example, in the case where the input data of the training data is a frontal fundus image, the output data may be an image in which labels have been given to regions such as the cup, the disc, and the peripheral portion of the optic nerve head. Note that the input data may be an image that has undergone image quality improvement, or may be an image that has not undergone image quality improvement.

[0270] Note that the label image used as output data may be an image in which labels have been given to the respective regions in the tomographic image by a physician or the like, or may be an image in which labels have been given to the respective regions by rule-based region detection processing. However, if a label image for which labels are not appropriately labeled is used as output data of training data for machine learning, there is a possibility that the image obtained using the learning model (which has been learned using the training data in question) will also be a label image for which labels are not appropriately labeled. Therefore, by excluding pairs including such label images from the training data, the possibility of generating inappropriate label images using the learning model can be reduced. Here, the term "rule-based region detection processing" refers to a detection process that utilizes, for example, known regularities (such as the regularity of the retinal shape).

[0271] By performing segmentation processing using such a learning model for segmentation, it can be expected that the image processing units 320, 1620, and 1920 will detect specific areas quickly and accurately for various images. Note that the learning model for segmentation can be used as the first learning model mentioned in Modification 3. In addition, in Embodiment 3, the analysis unit 1924 can perform segmentation processing using the learning model according to this modification.

[0272] Note that a learning model for segmentation may be prepared for each of the various images used as input data. In addition, the learning model for segmentation may be a learning model that has been learned for images of each imaging site (e.g., the center of the macular area and the center of the optic nerve head), or may be a learning model that has been learned regardless of the imaging site.

[0273] Furthermore, when generating an en-face image or an OCTA frontal image, a depth range is set and specified as described later. Therefore, for an en-face image or an OCTA frontal image, a learning model can be prepared for each depth range used to generate the image.

[0274] Note that, for at least one of the image before image quality improvement processing and the image after image quality improvement processing by the image quality improvement unit 322 or 1622, the image processing units 320, 1620, and 1920 can perform segmentation processing using a rule-based segmentation process or a learning model. Therefore, the image processing unit 320 can identify different regions in at least one of the images in question. In particular, the image processing units 320, 1620, and 1920 perform segmentation processing using a learning model (third learning model) for segmentation that is different from the learning model used to generate the high-quality image (the second medical image). Therefore, it can be expected that different regions in at least one of the images in question will be quickly and accurately identified.

[0275] (Variant 6)

[0276] The high-quality image obtained by the image quality improvement units 322 and 1622 using the learning model according to the above-described embodiments and variations can be manually modified according to instructions from the operator. For example, the image quality improvement model can be updated by incremental learning, which uses high-quality images in which image processing of a designated area is changed according to the inspector's instructions as training data. In this case, for example, an image modified in the following manner can be used as training data for incremental learning: in a high-quality image generated using the image quality improvement model, a grayscale conversion process is performed on the retina for an area that has been subjected to a grayscale conversion process for the vitreous body or the choroid. Conversely, an image modified in the following manner can be used as training data for incremental learning: in a high-quality image generated using the image quality improvement model, a grayscale conversion process is performed on the vitreous body or the choroid for an area that has been subjected to a grayscale conversion process for the retina.

[0277] In addition, the image quality improvement model can be updated by incremental learning, wherein the value of the ratio set (changed) according to the instruction from the inspector is adopted as learning data. For example, if the inspector tends to set the ratio of the input image to the high-quality image higher when the input image is relatively dark, the learning model performs incremental learning to have this tendency. Therefore, for example, the learning model can be customized to obtain a learning model that can obtain a combination ratio that matches the inspector's preference.

[0278] At this time, a button for determining whether to use the set (changed) value of the ratio as training data for incremental learning according to the instruction of the examiner may be displayed on the display screen. In this way, the control unit 30, 1600, or 1900 can determine whether incremental learning is required according to the instruction of the operator. In addition, the following configuration may be adopted: the ratio determined using the learning model is taken as a default value, and then the ratio value can be changed from the default value according to the instruction from the examiner.

[0279] Note that, as described above, the learning model may also be set in a device such as a server. In this case, the control unit 30, 1600, or 1900 may perform incremental learning according to the operator's instruction, and send and store the input image and the high-quality image to which the above-mentioned modification has been made as a pair of training data to a relevant server, etc. In other words, the control unit 30, 1600, or 1900 may determine whether to send the training data for incremental learning to a device such as a server equipped with a learning model according to the operator's instruction.

[0280] Please note that for the various learning models described in the aforementioned embodiments and other variations, each learning model can also similarly use data manually modified according to the operator's instructions as training data for incremental learning. In addition, it is also possible to determine whether incremental learning is required and whether to send data to the server by similar methods. In these cases as well, it can be expected that the accuracy of various processing will be improved, and processing according to the inspector's preference can be performed.

[0281] For example, for a learning model for segmentation, incremental learning can be performed in which data manually modified according to the operator's instructions is used as training data. In addition, whether incremental learning is required and whether data is sent to the server are determined by the same method as described above. In these cases as well, it can be expected that the accuracy of the segmentation processing will be improved, and processing according to the inspector's preference tendency can be performed.

[0282] (Variant 7)

[0283] In the above-mentioned embodiments and modifications, the image processing units 320, 1620 and 1920 can generate an en-face image or an OCTA frontal image of the eye to be inspected using a three-dimensional tomographic image. In this case, the display control unit 350 can display the generated en-face image or OCTA image on the display unit 50. In addition, the analysis unit 1924 can also analyze the generated en-face image or OCTA image.

[0284] The en-face image and the OCTA frontal image will now be described. The en-face image is a frontal image generated by projecting data on an arbitrary depth range in a three-dimensional tomographic image obtained using optical interference in the X and Y directions. The frontal image is generated by projecting or integrating data corresponding to a depth range, which is at least a local depth range of the volume data (three-dimensional tomographic image) obtained using optical interference, onto a two-dimensional plane and is defined based on two reference planes.

[0285] For example, an en-face image can be generated by projecting data corresponding to a depth range determined based on the retinal layer detected by segmentation processing for the two-dimensional tomographic image among the volume data onto a two-dimensional plane. Note that as a technique for projecting data corresponding to a depth range defined based on two reference planes onto a two-dimensional plane, for example, a technique for using a representative value of data within a relative depth range as a pixel value on a two-dimensional plane can be used. In this case, examples of representative values ​​may include values ​​such as an average, median, or maximum value of pixel values ​​within a range (depth range) in the depth direction of an area surrounded by two reference planes.

[0286] The depth range associated with the en-face image can be specified based on, for example, two layer boundaries associated with the retinal layer, which are detected by the above-mentioned technique using the rule-based segmentation process or the segmentation process using the learning model mentioned in Modification 5. In addition, with reference to one of the two layer boundaries associated with the retinal layer detected by these segmentation processing operations, the depth range in question may include only a range corresponding to a predetermined number of pixels in the deeper direction or the shallower direction. In addition, the depth range associated with the en-face image may be, for example, a range that has been changed (offset) from the range between the two layer boundaries associated with the detected retinal layer according to an instruction of the operator. At this time, for example, the operator can change the depth range by moving an indicator indicating the upper limit or lower limit of the depth range, which has been superimposed on the tomographic image for which image quality improvement has been performed by the image quality improvement unit 322 or 1622 or on the tomographic image for which image quality improvement has not yet been performed.

[0287] Note that, as described above, the generated en-face image is not limited to an en-face image based on a luminance value (luminance en-face image). The generated en-face image may be, for example, a motion contrast en-face image, which is generated by projecting or integrating data corresponding to the above-mentioned depth range on a two-dimensional plane for motion contrast data relative to a plurality of volume data items. Here, the term "motion contrast data" refers to data indicating changes between a plurality of volume data items obtained by controlling so that the measurement light is scanned multiple times on the same area (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, at each position different from each other, by obtaining data showing changes between a plurality of tomographic images obtained at approximately the same position, the motion contrast data can be obtained as volume data. Note that with respect to OCT angiography for measuring blood flow motion, the motion contrast en-face image is also referred to as an OCTA en-face image (OCTA en-face image), and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a variance value or decorrelation value between two tomographic images or between interference signals corresponding to the two tomographic images, or as a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value), and can be obtained by any known method. At this time, two tomographic images can be obtained, for example, by controlling so that the measurement light is scanned multiple times on the same area (same position) of the eye to be inspected.

[0288] In addition, three-dimensional OCTA data (OCT volume data) used when generating an OCTA en face image can be generated using at least a portion of the interference signal shared with the volume data including the tomographic image used for image segmentation. In this case, the volume data (three-dimensional tomographic image) and the three-dimensional OCTA data can correspond to each other. Therefore, for example, by using the three-dimensional motion contrast data corresponding to the volume data, a motion contrast en face image corresponding to a depth range determined based on the retinal layer detected by image segmentation can be generated.

[0289] Note that the volume data used when generating the en-face image or the OCTA frontal image may be composed of a tomographic image whose image quality has been improved by the image quality improvement unit 322 or 1622. In other words, the image processing unit 320, 1620, or 1920 may generate the en-face image or the OCTA frontal image using the volume data composed of a plurality of tomographic images obtained at a plurality of different positions whose image quality has been improved. In other words, in the case where the images before and after the image quality improvement process by the image quality improvement unit 322 or 1622 are three-dimensional OCT tomographic images, the image processing unit 320, 1620, or 1920 may generate a frontal image corresponding to the local depth range of the image after the image quality improvement process. Therefore, the image processing unit 320, 1620, or 1920 may generate a high-quality frontal image based on the high-quality three-dimensional tomographic image.

[0290] (Variant 8)

[0291] Next, we will refer to FIG. 21A to FIG. 23 An image processing device according to modification 8 is described. In the above-described embodiments and modifications, the image quality improvement units 322 and 1622 perform image quality improvement processing for tomographic images using a learning model (image quality improvement model) for improving image quality. In this regard, the image quality improvement units 322 and 1622 may perform image quality improvement processing using image quality improvement models for other images, and the display control unit 350 may cause various images for which image quality has been improved to be displayed on the display unit 50. For example, the image quality improvement unit 322 or 1622 may cause a luminance en-face image or an OCTA frontal image, etc. to undergo image quality improvement processing. In addition, the display control unit 350 may cause the display unit 50 to display at least one of a tomographic image, a luminance en-face image, and an OCTA frontal image that have undergone image quality improvement processing by the image quality improvement unit 322 or 1622. Note that the image that undergoes image quality improvement and is displayed may be an SLO fundus image, a fundus image obtained using a fundus camera (not shown), or a fluorescent fundus image, etc.

[0292] In this case, similar to the training data of the image quality improvement model according to the above-mentioned embodiment and the modification, for the training data of the image quality improvement model for performing image quality improvement processing on various images, for various images, the image before the image quality improvement processing is used as input data, and the image after the image quality improvement processing is used as output data. Note that, for the image quality improvement processing related to the training data, similar to the above-mentioned embodiment and the modification, the processing may be, for example, arithmetic mean processing, processing using a smoothing filter, maximum a posteriori processing (MAP estimation processing), or grayscale conversion processing. In addition, as the image after the image quality improvement processing, for example, an image to which filtering processing such as noise removal and edge enhancement is performed may be used, or an image to which contrast is adjusted so as to obtain a high-brightness image from a low-brightness image may be used. In addition, for the output data of the training data related to the image quality improvement model, since it is sufficient for the output data to be a high-quality image, the output data may be an image photographed using an OCT device with higher performance than the OCT device used to photograph the image as the input data, or may be an image photographed with a high load setting.

[0293] Furthermore, image quality improvement models may be prepared for various images to be subjected to image quality improvement processing. For example, an image quality improvement model for tomographic images, an image quality improvement model for luminance en-Face images, and an image quality improvement model for OCTA frontal images may be prepared. In addition, the image quality improvement model for luminance en-Face images or the image quality improvement model for OCTA frontal images may be a learning model that has comprehensively learned images at different depth ranges for a depth range (generation range) related to the generation of images. Images at different depth ranges may include, for example, Fig.21A The images of the surface layer (Im2110), deep layer (Im2120), outer layer (Im2130), and choroidal vascular network (Im2140) are shown. In addition, for the image quality improvement model for the luminance en-face image or the image quality improvement model for the OCTA frontal image, a plurality of image quality improvement models for learning images of different depth ranges can be prepared. Note that the image quality improvement model for performing image quality improvement processing on images other than tomographic images is not limited to an image quality improvement model for performing different image processing on each region, and may be an image quality improvement model for performing the same image processing on the entire image.

[0294] Furthermore, in the case of preparing an image quality improvement model for a tomographic image, the image quality improvement model may be a learning model that comprehensively learns tomographic images obtained at different positions in the sub-scanning direction (Y-axis direction). Fig.21BThe tomographic images Im2151 to Im2153 shown are examples of tomographic images obtained at different positions in the sub-scanning direction. However, in the case of images obtained by imaging different positions of an imaging part (for example, the center of the macular area and the center of the optic nerve head), a structure may be adopted so that learning is performed separately for each imaging part, or a structure may be adopted so that learning is performed together without paying attention to the imaging part. Note that a luminance tomographic image and a tomographic image of motion contrast data may be included as tomographic images for which image quality improvement is to be performed. However, since the image feature values ​​are significantly different between the luminance tomographic image and the tomographic image of motion contrast data, learning may be performed separately as individual image quality improvement models for various images.

[0295] In the present modification, an example is described in which the display control unit 350 displays the image on which the image quality improvement processing is performed by the image quality improvement unit 322 or 1622 on the display unit 50. Note that although Fig.22A and Fig. 22B This modification example is described, but the display screen is not limited to Fig.22A and Fig. 22B The example shown. The image quality improvement process (quality improvement process) can also be similarly applied to a display screen that displays multiple images obtained at different dates and times side by side, as in the case of a follow-up observation. In addition, as in the case of the imaging confirmation screen, the image quality improvement process can also be similarly applied to a display screen where the inspector confirms whether the imaging is successful immediately after imaging. The display control unit 350 can cause multiple high-quality images generated by the image quality improvement unit 322 or 1622 or low-quality images for which image quality improvement has not been performed to be displayed on the display unit 50. In addition, for the multiple high-quality images or low-quality images for which image quality improvement has not been performed displayed on the display unit 50, the display control unit 350 can cause each of the low-quality images and high-quality images selected according to the inspector's instructions to be displayed on the display unit 50. The image processing device can also output the low-quality images and high-quality images selected according to the inspector's instructions to the outside of the image processing device.

[0296] In the following, reference is made to Fig.22A and Fig. 22BAn example of a display screen 2200 of an interface according to the present modification is described. The display screen 2200 represents the entire screen, and a "Patient" tab 2201, an "Imaging" tab 2202, a "Report" tab 2203, and a "Setting" tab 2204 are shown in the display screen 2200. In addition, a diagonal line in the "Report" tab 2203 indicates an activation state of the report screen. In the present modification, an example of displaying a report screen will be described.

[0297] exist Fig.22A In the report screen shown in , an SLO fundus image Im2205, OCTA front images Im2207 and Im2208, a luminance en-face image Im2209, tomographic images Im2211 and Im2212, and a button 2220 are shown. In addition, on the SLO fundus image Im2205, an OCTA front image Im2206 corresponding to the OCTA front image Im2207 is displayed in a superimposed manner. In addition, on the tomographic images Im2211 and Im2212, boundary lines 2213 and 2214 of the depth ranges of the OCTA front images Im2207 and Im2208 are respectively displayed in a superimposed manner. The button 2220 is a button for designating the execution of the image quality improvement process. As described later, the button 2220 may be a button for inputting an instruction to display a high-quality image.

[0298] In the present modification, execution of the image quality improvement process is performed when the button 2220 is designated, or whether to perform the image quality improvement process is determined based on information stored (saved) in the database. First, an example of switching between display of a high-quality image and display of a low-quality image by designating the button 2220 according to an instruction from the examiner will be described. Hereinafter, the OCTA frontal image will be described as a target image for the image quality improvement process.

[0299] Note that the depth range of the OCTA frontal images Im2207 and Im2208 can be determined using information on the retinal layers detected by the above-mentioned conventional segmentation processing or by the segmentation processing using the learning model. The depth range may be, for example, a range between two layer boundaries associated with the detected retinal layers, or may refer to a range corresponding to only a predetermined number of pixels in the deeper direction or the shallower direction of one of the two layer boundaries associated with the detected retinal layers. In addition, the depth range may be a range that has been changed (shifted) from the range between the two layer boundaries associated with the detected retinal layers according to, for example, an instruction of an operator.

[0300] When the examiner designates the "Report" tab 2203 to transition to the report screen, the display control unit 350 displays the low image quality OCTA frontal images Im2207 and Im2208. Thereafter, when the examiner designates the button 2220, the image quality improvement unit 322 or 1622 performs image quality improvement processing on the OCTA frontal images Im2207 and Im2208 displayed on the screen. After the image quality improvement processing is completed, the display control unit 350 displays the high-quality image generated by the image quality improvement unit 322 or 1622 on the report screen. Note that since the OCTA frontal image Im2206 is an image in which the OCTA frontal image Im2207 is displayed in a superimposed manner on the SLO fundus image Im2205, the display control unit 350 can also cause an image obtained by performing image quality improvement processing on the OCTA frontal image Im2206 to be displayed. Furthermore, the display control unit 350 may change the display of the button 2220 to an activated state to provide a display from which it can be understood that the image quality improvement process is performed.

[0301] In this case, the execution of the processing by the image quality improvement unit 322 or 1622 does not have to be limited to the timing of the examiner specifying button 2220. Since the types of OCTA frontal images Im2207 and Im2208 to be displayed when the report screen is opened are known in advance, the image quality improvement unit 322 or 1622 can perform image quality improvement processing when the displayed screen is switched to the report screen. Subsequently, at the timing of pressing button 2220, the display control unit 350 can display a high-quality image on the report screen. In addition, the number of types of images that are subjected to image quality improvement processing in response to an instruction from the examiner or when switching to the report screen does not have to be two. Such a construction may also be adopted to process images with a high probability of being displayed, for example, images such as Fig.21A In this case, the image obtained by performing the image quality improvement processing may be temporarily stored in a memory or may be stored in a database.

[0302] Next, a case where the image quality improvement process is performed based on the information stored (saved) in the database is described. In the case where the state of execution of the image quality improvement process to be performed is stored in the database, when the display is switched to the report screen, the display control unit 350 causes the high-quality image obtained as a result of the image quality improvement process performed by the image quality improvement unit 322 or 1622 to be displayed on the display unit 50 by default. In addition, such a configuration may be adopted in which the button 2220 is displayed in an activated state by default by the display control unit 350, so that the inspector can know that the high-quality image obtained by performing the image quality improvement process is being displayed. If the inspector desires to display a low-quality image in the state before the image quality improvement process, the inspector can perform an operation of designating the button 2220, thereby releasing the activated state, so that the display control unit 350 causes the low-quality image to be displayed on the display unit 50. At this time, if the inspector desires to return the displayed image to a high-quality image, the inspector designates the button 2220 so as to enter the activated state, whereby the display control unit 350 causes the high-quality image to be displayed on the display unit 50 again.

[0303] It is assumed that it is possible to collectively specify for all data stored in a database whether to perform image quality improvement processing on the data stored in the database for each category of data, such as each group of video data (each inspection). For example, in a case where a state has been stored in which image quality improvement processing is to be performed on the entire database, the inspector can store a state in which image quality improvement processing is not performed on a single video data item (personal inspection). In this case, for a single video data item for which a state in which image quality improvement processing is not performed is stored, the next time the relevant video data is displayed, the video data can be displayed in a state in which image quality improvement processing has not yet been performed on it. According to this construction, in a case in which it is not specified whether to perform image quality improvement processing in a video data unit (inspection unit), processing can be performed based on information specified for the entire database. In addition, in a case in which it has been specified that image quality improvement processing is to be performed in a video data unit (inspection unit), processing can be performed individually based on the information in question.

[0304] Note that a user interface (not shown) (e.g., a "Store" button) can be used to store a state in which image quality improvement processing has been performed for each imaging data item (each examination). In addition, when switching to other imaging data (another examination) or other patient data (e.g., changing to a display screen other than a report screen according to an instruction from the examiner), based on the display state (e.g., the state of button 2220), a state in which image quality improvement processing is to be performed can be stored.

[0305] Although an example in which the OCTA frontal images Im2207 and Im2208 are displayed as OCTA frontal images is shown in this modification, the OCTA frontal images to be displayed can be changed according to the inspector's designation. Therefore, a description will now be given of changing the displayed image in the case where execution of the image quality improvement process has been designated (button 2220 is in an activated state).

[0306] The change of the displayed image is performed using a user interface (not shown) (e.g., a combo box). For example, in the case where the examiner changes the type of the image from the surface layer image to the choroidal vascular network image, the image quality improvement unit 322 or 1622 performs image quality improvement processing on the choroidal vascular network image, and the display control unit 350 displays the high-quality image generated by the image quality improvement unit 322 or 1622 on the report screen. In other words, in response to an instruction from the examiner, the display control unit 350 can change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range that is at least partially different from the first depth range. At this time, according to changing the first depth range to the second depth range in response to an instruction from the examiner, the display control unit 350 can change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range. Please note that, in the case where a high-quality image has been generated for an image that is likely to be displayed when the report screen is transitioned to, as described above, the display control unit 350 can display the high-quality image that has been generated.

[0307] In addition, the method for changing the image type is not limited to the above method, and it is also possible to generate an OCTA frontal image for which a different depth range is set by changing the layer and offset value used as a reference, and display a high-quality image obtained by performing image quality improvement processing on the generated OCTA frontal image. In this case, when the layer or offset value used as a reference is changed, the image quality improvement unit 322 or 1622 performs image quality improvement processing for any OCTA frontal image, and the display control unit 350 displays the high-quality image on the report screen. Note that the change of the layer or offset value used as a reference can be made using a user interface (not shown) (e.g., a combo box or a text box). In addition, the depth range (generation range) of the OCTA frontal image can also be changed by dragging any one of the boundary lines 2213 and 2214 (moving the layer boundary) displayed in a superimposed manner on the tomographic images Im2211 and Im2212, respectively.

[0308] In the case where the boundary line is changed by dragging, the execution command for the image quality improvement process is continuously executed. Therefore, the image quality improvement unit 322 or 1622 may be configured to always perform the process for the execution command, or may be configured to perform the process after the layer boundary has been changed by dragging. Alternatively, although the command to execute the image quality improvement process is continuously issued, the image quality improvement unit 322 or 1622 may be configured to cancel the previous command at the time point when the next command arrives, and execute the latest command.

[0309] Note that in some cases, the image quality improvement process takes a relatively long time. Therefore, even when the command is executed at any timing in the above timing, it may take a relatively long time until the high-quality image is displayed. Therefore, during the period from the depth range for generating the OCTA frontal image in response to the instruction from the examiner to the display of the high-quality image, a low-quality OCTA frontal image (low-quality image) corresponding to the set depth range can be displayed. In other words, the following configuration can be adopted: when the above-mentioned depth range is set, a low-quality OCTA frontal image (low-quality image) corresponding to the set depth range is displayed, and when the image quality improvement process is completed, the display of the low-quality OCTA frontal image is changed to the display of the high-quality image. In addition, information indicating that the image quality improvement process is being performed can be displayed during the period from the setting of the above-mentioned depth range to the display of the high-quality image. Note that these processing operations are not limited to the configuration applied in the following case: the state is a state in which the execution of the image quality improvement process has been specified (the button 2220 is in the active state). For example, these processing operations can also be applied during the period until the high-quality image is displayed when the image quality improvement process is instructed to be performed according to the instruction from the examiner.

[0310] Although an example is shown in the present modification in which OCTA frontal images Im2207 and Im2208 related to different layers are displayed as OCTA frontal images, and a low-quality image and a high-quality image are displayed by switching therebetween, the displayed image is not limited thereto. For example, a configuration may be adopted whereby a low-quality OCTA frontal image is displayed side by side as the OCTA frontal image Im2207 and a high-quality OCTA frontal image is displayed side by side as the OCTA frontal image Im2208. In the case of displaying images by switching therebetween, since the images are switched at the same place, it is easy to compare the portions where changes have occurred, and in the case of displaying the images side by side, since the images can be displayed at the same time, it is easy to compare the entire images.

[0311] Next, Fig.22A and Fig. 22B The execution of the image quality improvement process in the case of screen transition will be described. Fig. 22B It is displayed in an enlarged manner Fig.22A An example of a screen of the OCTA frontal image Im2207 is shown. Fig. 22B In the example screen shown, something like Fig.22A , button 2220 is displayed. For example, by double-clicking the OCTA frontal image Im2207, specifying Fig.22A The screen shown to Fig. 22B The screen transition of the screen shown in FIG. 22 is displayed, and the screen transition from the screen shown in FIG. 22 is displayed ... by clicking the "Close" button 2230. Fig. 22B The screen shown to Fig.22A Note that, for screen transition, the method for transitioning from one screen to another is not limited to the method described here, and a user interface (not shown) may also be used.

[0312] When the execution of the image quality improvement process has been designated at the time of screen transition (button 2220 is in the activated state), this state is maintained even when the screen transition occurs. Fig. 22B The screen shown changes to Fig.22A In the case where a high-quality image is displayed on the screen shown in the figure, the high-quality image is also displayed on the Fig. 22B In addition, button 2220 is activated. The same applies to Fig. 22B The screen changes to Fig.22A The screen shown in the figure is Fig. 22B On the screen shown, the display can also be switched to a low-quality image by specifying button 2220.

[0313] For screen transition, screen transition is not limited to the screen described here, and as long as it is transitioned to a screen displaying the same camera data, for example, a display screen for tracking observation or a display screen for panoramic images, the transition can be performed while maintaining the display state of the high-quality image. In other words, on the display screen after the transition, an image corresponding to the state of the button 2220 on the display screen before the transition can be displayed. For example, if the button 2220 is in an activated state on the display screen before the transition, a high-quality image is displayed on the display screen after the transition. In addition, for example, if the activation state of the button 2220 is released on the display screen before the transition, a low-quality image is displayed on the display screen after the transition. Note that the following configuration may be adopted: if the button 2220 is in an activated state on the display screen for tracking observation, a plurality of images displayed side by side on the display screen for tracking observation obtained at different dates and times (different inspection days) are switched to high-quality images. In other words, the following configuration may be adopted: if the button 2220 is in an activated state on the display screen for tracking observation, switching to high-quality images is performed collectively for a plurality of images obtained at different dates and times.

[0314] exist Fig.23 An example of a display screen for tracking observation is shown in FIG. When tab 2301 is selected in response to an instruction from the examiner, the following is displayed: Fig.23 The display screen for tracking observation shown in FIG. 230 is a diagram showing a display screen for tracking observation. At this time, the depth range of the OCTA frontal image can be changed by the examiner by selecting a desired set from the predetermined depth range sets displayed in list boxes 2302 and 2303. For example, surface capillaries are selected in list box 2302, and deep capillaries are selected in list box 2303. The analysis results of the OCTA frontal image of the surface capillaries are displayed in the upper display area, and the analysis results of the OCTA frontal image of the deep capillaries are displayed in the lower display area. When the depth range is selected, a plurality of images acquired at different dates and times are collectively changed to a parallel display of the analysis results of the plurality of OCTA frontal images within the selected depth range.

[0315] At this time, if the display of the analysis results is placed in an unselected state, the display can be collectively changed to a parallel display of the analysis results of the plurality of OCTA frontal images acquired at different dates and times. If the button 2220 is then designated according to an instruction from the examiner, the display of the plurality of OCTA frontal images is collectively changed to a display of a plurality of high-quality images.

[0316] In addition, when the display of the analysis results is in the selected state, if the button 2220 is designated according to the instruction from the examiner, the display of the analysis results of the plurality of OCTA frontal images is collectively changed to the display of the analysis results of the plurality of high-quality images. Here, the display of the analysis results may be performed so that the analysis results are displayed on the image in a superimposed manner with any transparency. At this time, the change from the display of the image to the display of the analysis results may be, for example, a change to a state in which the analysis results are superimposed on the image being displayed with any transparency. In addition, the change from the display of the image to the display of the analysis results may be, for example, a change to the display of an image (e.g., a two-dimensional graph) obtained by fusing the respective analysis results and the image with any transparency.

[0317] In addition, such layer boundaries and offset positions for specifying the depth range can be changed collectively from user interfaces 2305 and 2306, respectively. Note that user interfaces 2305 and 2306 are one example of an interface for changing the type and offset position of layer boundaries, and any other form of interface may also be used. Note that the depth ranges of a plurality of OCTA frontal images acquired on different dates and times can be collectively changed in accordance with instructions from the examiner by also displaying a tomographic image together with it and moving the layer boundary data superimposed on the tomographic image. At this time, a plurality of tomographic images acquired on different dates and times can be displayed side by side, and when the above-mentioned movement is performed on one tomographic image, the layer boundary data can be similarly moved on the other tomographic images.

[0318] Furthermore, for example, the image projection method and whether or not to perform projection artifact removal processing can be changed by selecting from a user interface such as a context menu.

[0319] In addition, the selection button 2307 can be selected to display a selection screen (not shown), and an image selected from the image list displayed on the selection screen can be displayed. Fig.23 The arrow 2304 displayed in the upper portion of the illustrated screen is a mark indicating the currently selected examination, and the reference examination (baseline) is the examination selected in subsequent imaging ( Fig.23 Naturally, a mark indicating the reference check may be displayed on the display unit.

[0320] Furthermore, in the case where the "Show Difference" check box 2308 is specified, the measurement value distribution (graph or sector graph) regarding the reference image is displayed on the reference image. In addition, in this case, in the area corresponding to the inspection date other than the inspection date of the reference image, a differential measurement value graph is displayed, which shows the difference between the measurement value distribution calculated for the reference image and the measurement value distribution calculated for the image displaying the relevant area. As the measurement result, a trend curve graph (a curve graph of the measurement values ​​for the images of each inspection date obtained by measuring the change over time) can be displayed on the report screen. In other words, time series data (for example, a time series graph) of multiple analysis results corresponding to multiple images obtained at different dates and times can be displayed. At this time, with respect to the analysis results related to the dates and times other than the multiple dates and times corresponding to the multiple images displayed, the analysis results can also be displayed as time series data in the following state: the analysis results can be distinguished from the multiple analysis results corresponding to the multiple images being displayed (for example, the color of each point on the time series graph is different depending on whether the corresponding image is displayed). In addition, the regression line (curve) of the trend graph and the corresponding mathematical formula can be displayed on the report screen.

[0321] Although a description related to the OCTA en-face image has been given in the present modification, the image to which the processing according to the present modification is applied is not limited thereto. The image related to the processing for displaying an image, image quality improvement, and image analysis, etc. according to the present modification may be a luminance en-face image. In addition, different kinds of images are not limited to en-face images, and may be different images such as a tomographic image obtained by a B-scan, an SLO fundus image, a fundus image, or a fluorescent fundus image. In this case, the user interface for performing the image quality improvement processing may be a user interface for instructing the execution of the image quality improvement processing for a plurality of images of different kinds, or may be a user interface for selecting any image from a plurality of images of different kinds and instructing the execution of the image quality improvement processing.

[0322] For example, when improving the image quality of a tomographic image obtained by B scanning and displaying the resulting image, it is possible to Fig.22AThe tomographic images Im2211 and Im2212 shown are subjected to image quality improvement and displayed. In addition, tomographic images subjected to image quality improvement can be displayed in the area where the OCTA frontal images Im2207 and Im2208 are displayed. Note that the number of tomographic images subjected to image quality improvement and displayed may be one tomographic image or may be a plurality of tomographic images. In the case of displaying a plurality of tomographic images, tomographic images obtained at positions different from each other in the sub-scanning direction may be displayed, and, for example, in the case of performing image quality improvement and displaying a plurality of tomographic images obtained by cross scanning or the like, respective images in different scanning directions may be displayed. Furthermore, for example, in the case of performing image quality improvement and displaying a plurality of tomographic images obtained by radial scanning or the like, some selected (multiple) tomographic images (for example, two tomographic images at positions symmetrical to each other with respect to a reference line) may be displayed individually. In addition, as Fig.23 As shown, a plurality of tomographic images can be displayed on a display screen for tracking observation, and instructions for image quality improvement and analysis results (e.g., thickness of a specific layer) can be displayed by a method similar to the above method. In addition, image quality improvement processing can be performed on tomographic images based on information stored in a database by the same method as the above method.

[0323] Similarly, in the case of performing image quality improvement on an SLO fundus image and displaying the resulting image, for example, the image quality of the SLO fundus image Im2205 may be improved and displayed. Also, in the case of performing image quality improvement on a luminance en-face image and displaying the resulting image, for example, the image quality of the luminance en-face image Im2209 may be improved and displayed. Fig.23 As shown, multiple SLO fundus images and en-face images can be displayed on a display screen for tracking observation, and instructions and analysis results (e.g., the thickness of a specific layer) for image quality improvement can be displayed by a method similar to the above method. In addition, image quality improvement processing can be performed on SLO fundus images and en-face images based on information stored in a database by the same method as the above method. Please note that the display of tomographic images, SLO fundus images, and luminance en-face images is for demonstration purposes, and these images can be displayed in any form according to the desired construction. In addition, at least two or more of the OCTA frontal images, tomographic images, SLO fundus images, and luminance en-face images can be improved in image quality and displayed based on a single instruction.

[0324] According to the aforementioned configuration, the display control unit 350 can display the image subjected to the image improvement processing by the image quality improvement unit 322 or 1622 of the present modification on the display unit 50. At this time, as described above, in the case of a state where at least one condition is selected among a plurality of conditions related to the display of a high-quality image, the display of an analysis result, the depth range of a front image to be displayed, etc., even if the display screen is changed to another display screen, the selected state can be maintained.

[0325] In addition, as described above, in the case where the state of at least one condition among the multiple conditions is selected, even if another condition is changed to the selected state, the state of selecting at least one condition can be maintained. For example, in the case where the display of the analysis result is in the selected state, in response to an instruction from the inspector (for example, when the button 2220 is designated), the display control unit 350 can change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image. In addition, in the case where the display of the analysis result is in the selected state, in response to an instruction from the inspector (for example, when the designation of the button 2220 is released), the display control unit 350 can change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image.

[0326] In addition, in the case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the designation of the display of the analysis result is released), the display control unit 350 may change the display of the analysis result of the low-quality image to the display of the low-quality image. In addition, in the case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the display of the analysis result is specified), the display control unit 350 may change the display of the low-quality image to the display of the analysis result of the low-quality image. In addition, in the case where the display of the high-quality image is in a selected state, in response to an instruction from the inspector (for example, when the designation of the display of the analysis result is released), the display control unit 350 may change the display of the analysis result of the high-quality image to the display of the high-quality image. In addition, in the case where the display of the high-quality image is in a selected state, in response to an instruction from the inspector (for example, when the display of the analysis result is specified), the display control unit 350 may change the display of the high-quality image to the display of the analysis result of the high-quality image.

[0327] Furthermore, let us consider a case where the display of the high-quality image is in an unselected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second analysis result is specified), the display control unit 350 may change the display of the first analysis result of the low-quality image to the display of the second analysis result of the low-quality image. Furthermore, let us consider a case where the display of the high-quality image is in a selected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second analysis result is specified), the display control unit 350 may change the display of the first analysis result of the high-quality image to the display of the second analysis result of the high-quality image.

[0328] Note that the following configuration may be adopted: on the display screen for tracking observation, as described above, for a plurality of images obtained at different dates and times, changes in these displays are collectively reflected. Here, the display of the analysis results may be performed so that the analysis results are displayed on the image in a superimposed manner with any transparency. At this time, the display changed to the analysis results may be, for example, changed to a state in which the analysis results are superimposed on the image being displayed with any transparency. In addition, the display changed to the analysis results may be, for example, changed to a display of an image (e.g., a two-dimensional graph) obtained by performing fusion processing of the respective analysis results and images with any transparency.

[0329] Note that in this modification, the image quality improvement unit 322 or 1622 generates a high-quality image by improving the image quality of the tomographic image using an image quality improvement model. However, the component that generates a high-quality image using an image quality improvement model is not limited to the image quality improvement units 322 and 1622. For example, a second image quality improvement unit separated from the image quality improvement units 322 and 1622 may be provided, and the second image quality improvement unit may generate a high-quality image using an image quality improvement model. In this case, the second image quality improvement unit may generate a high-quality image in which the same image processing is performed on the entire image, rather than a high-quality image in which different image processing is performed on each region using a learning model. At this time, the output data of the learning model may be an image obtained by performing the same image quality improvement processing on the entire image. Note that the second image quality improvement unit or the image quality improvement model used by the second image quality improvement unit may be composed of a software module executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or may be composed of a circuit (e.g., an ASIC) that performs a specific function.

[0330] (Variant 9)

[0331] The display control unit 350 can cause the display unit 50 to display an image selected from the high-quality image generated by the image quality improvement unit 322 or 1622 and the input image according to an instruction from the inspector. In addition, the display control unit 350 can switch the display on the display unit 50 from the camera image (input image) to the high-quality image according to an instruction from the inspector. In other words, the display control unit 350 can change the display of the low-quality image to the display of the high-quality image according to an instruction from the inspector. In addition, the display control unit 350 can change the display of the high-quality image to the display of the low-quality image according to an instruction from the inspector.

[0332] Furthermore, the image quality improvement unit 322 or 1622 can start image quality improvement processing (input the image to the image quality improvement engine) using the image quality improvement model according to the instruction from the examiner, and the display control unit 350 can display the generated high-quality image on the display unit 50. In contrast, when the imaging device (imaging unit 20) captures an input image, the image quality improvement unit 322 or 1622 can automatically generate a high-quality image based on the input image by using the image quality improvement model, and the display control unit 350 can cause the display unit 50 to display the high-quality image according to the instruction from the examiner.

[0333] Note that these processes may also be similarly performed on the output of the analysis results. In other words, the display control unit 350 may change the display of the analysis results of the low-quality image to the display of the analysis results of the high-quality image according to the instruction from the inspector. In addition, the display control unit 350 may change the display of the analysis results of the high-quality image to the display of the analysis results of the low-quality image according to the instruction from the inspector. In addition, the display control unit 350 may change the display of the analysis results of the low-quality image to the display of the low-quality image according to the instruction from the inspector. In addition, the display control unit 350 may change the display of the analysis results of the low-quality image to the display of the analysis results of the low-quality image according to the instruction from the inspector. In addition, the display control unit 350 may change the display of the analysis results of the high-quality image to the display of the high-quality image according to the instruction from the inspector. In addition, the display control unit 350 may change the display of the analysis results of the high-quality image to the display of the analysis results of the high-quality image according to the instruction from the inspector.

[0334] In addition, the display control unit 350 can change the display of the analysis result of the low-quality image to the display of another analysis result of the low-quality image according to the instruction from the examiner. In addition, the display control unit 350 can change the display of the analysis result of the high-quality image to the display of another analysis result of the high-quality image according to the instruction from the examiner.

[0335] Here, in the display of the analysis result of the high-quality image, the analysis result of the high-quality image may be superimposed and displayed on the high-quality image with an arbitrary transparency. In addition, in the display of the analysis result of the low-quality image, the analysis result of the low-quality image may be superimposed and displayed on the low-quality image with an arbitrary transparency. At this time, the change in the display of the analysis result may be, for example, a change in the state of superimposing the analysis result on the displayed image with an arbitrary transparency. In addition, the change in the display of the analysis result may be, for example, a change in displaying an image (for example, a two-dimensional graph) obtained by performing a fusion process on the analysis result and the image with an arbitrary transparency.

[0336] Note that in this modification, the image quality improvement units 322 and 1622 generate a high-quality image obtained by improving the image quality of the tomographic image using an image quality improvement model. However, the component that generates a high-quality image by using an image quality improvement model is not limited to the image quality improvement unit 322 or 1622. For example, a second image quality improvement unit different from the image quality improvement units 322 and 1622 may be provided, and the second image quality improvement unit may generate a high-quality image by using an image quality improvement model. In this case, the second image quality improvement unit may generate a high-quality image in which the same image processing is performed on the entire image, rather than a high-quality image in which different image processing is performed on each region using a learning model. At this time, the output data of the learning model may be an image obtained by performing the same image quality improvement processing on the entire image. Note that the second image quality improvement unit or the image quality improvement model used by the second image quality improvement unit may be composed of a software module executed by a processor such as a CPU, an MPU, a GPU, or an FPGA, or may be composed of a circuit (e.g., an ASIC) that performs a specific function.

[0337] In addition, in modification 8, an image on which image quality improvement processing has been performed by using the image quality improvement model is displayed according to the activation state of the button 2220 in the display screen. In contrast, the system may be configured so that, according to the activation state of the button 2220, an analysis value of the result of the segmentation processing using the learning model is displayed. In this case, for example, when the button 2220 is in an inactive state (the segmentation processing using the learning model is in an unselected state), the display control unit 350 causes the display unit 50 to display the analysis result using the result of the segmentation processing. In contrast, when the button 2220 becomes an activated state, the display control unit 350 causes the display unit 50 to display the analysis result using the result of the segmentation processing using the learning model.

[0338] In such a configuration, the analysis result using the result of the segmentation processing without using the learning model and the analysis result using the result of the segmentation processing using the learning model are switched and displayed according to the activation state of the button. Since these analysis results are respectively based on the result of the processing performed by the learning model and the result of the image processing performed by the rule base, the two results may be different. Therefore, by switching and displaying these analysis results, the examiner can compare the two results and can make a diagnosis using a more convincing analysis result.

[0339] Note that when the segmentation processing is switched, for example, in the case where the displayed image is a tomographic image, the numerical value of the layer thickness analyzed for each layer can be switched and displayed. In addition, for example, when a tomographic image divided into layers by color, hatching pattern, etc. is displayed, a tomographic image in which the shape of the layer is changed according to the result of the segmentation processing can be switched and displayed. In addition, when a thickness map is displayed as an analysis result, a thickness map in which the color indicating the thickness is changed according to the result of the segmentation processing can be displayed. In addition, the button for specifying the image quality improvement processing and the button for specifying the segmentation processing using the learning model can be set separately, only one button can be set, or both buttons can be set as one button.

[0340] In addition, similar to the switching of the above-mentioned image quality improvement processing, the switching of the segmentation processing can be performed based on the information stored (recorded) in the database. Note that for the processing at the time of screen transition, the switching of the segmentation processing can also be performed similarly to the above-mentioned image quality improvement processing.

[0341] (Variant 10)

[0342] The display control unit 350 in the various embodiments and modifications described above can cause analysis results such as the thickness of a desired layer or various blood vessel densities to be displayed on a report screen of a display screen. In addition, as analysis results, parameter values ​​(distributions) related to a site of interest can be displayed, the site including at least one of the optic nerve head, the macular region, the vascular region, the nerve bundle, the vitreous region, the macular region, the choroid region, the scleral region, the lamina cribrosa region, the retinal layer boundary, the retinal layer boundary edge, the photoreceptor cell, the blood cell, the vascular wall, the vascular inner wall boundary, the vascular outer boundary, the ganglion cell, the corneal region, the horn region, and the Schlemm's canal. At this time, for example, accurate analysis results can be displayed by analyzing a medical image that has been subjected to various artifact removal processes. Note that an artifact may be, for example, a false image region caused by light absorption by a vascular region or the like, a projection artifact, or a band-like artifact in a frontal image that appears in the main scanning direction of the measurement light due to the state of the subject's eye (movement or blinking, etc.). In addition, the artifact may be of any kind as long as the artifact is an imaging fault area, for example, the imaging fault area randomly appears on the medical image of a predetermined part of the subject each time imaging is performed. In addition, the display control unit 350 can cause the value (distribution) of a parameter related to an area including at least one of the above-mentioned various artifacts (imaging fault areas) to be displayed on the display unit 50 as an analysis result. In addition, the value (distribution) of a parameter related to an area including at least one abnormal part such as drusen, new blood vessel parts, leukocytes (hard exudate), pseudoacne, etc. can be displayed as an analysis result. Please note that the image analysis processing can be performed by the analysis unit 1924, or can be performed by an analysis unit separate from the analysis unit 1924. In addition, the image subjected to image analysis can be an image subjected to image quality improvement, or can be an image not subjected to image quality improvement.

[0343] The analysis result may be displayed using an analysis graph or using sectors indicating statistical values ​​corresponding to respective divided regions, etc. Note that the analysis result may be generated by the analysis unit 1924 or another analysis unit using a learning model (analysis result generation engine, or a learning model for generating analysis results) obtained by learning the analysis result of a medical image as training data. At this time, the learning model may be a model obtained by learning using training data including a medical image and the analysis result of the medical image, or training data including a medical image and the analysis result of a medical image of a type different from that of a related medical image, etc.

[0344] In addition, the training data may include a region label image generated by the segmentation process and a result of analyzing a medical image using the region label image. In this case, the image processing units 320, 1620, and 1920 can be used as an example of an analysis result generating unit, for example, which generates an analysis result for a tomographic image from a result obtained by performing the segmentation process using a learning model for generating an analysis result (e.g., a detection result of a retinal layer). In other words, by using a learning model (fourth learning model) for generating an analysis result that is different from a learning model for generating a high-quality image (second medical image), the image processing units 320, 1620, and 1920 can generate image analysis results for each different region identified by the segmentation process.

[0345] In addition, the learning model may be a model obtained by learning using training data including input data in which a plurality of medical images of different kinds of a predetermined part such as a luminance en-face image and a motion contrast en-face image are taken as a set. Here, the luminance en-face image corresponds to the luminance en-face image, and the motion contrast en-face image corresponds to the OCTA en-face image.

[0346] Furthermore, a configuration may be adopted so as to display an analysis result obtained using a high-quality image generated using a learning model for improving image quality. In this case, the input data included in the training data may be a high-quality image generated using a learning model for improving image quality, or may be a set consisting of a low-quality image and a high-quality image. Note that the training data may be an image obtained by manually or automatically modifying at least a portion of an image whose image quality has been improved using a learning model.

[0347] In addition, the training data may be, for example, data obtained by marking (annotating) input data for which information including at least one of an analysis value (e.g., an average value or a median value) obtained by analyzing an analysis area, a table including analysis values, an analysis graph, and the position of an analysis area such as a sector in an image, etc. is adopted as correct answer data (for supervised learning). Note that a configuration may be adopted in which the analysis result obtained using the learning model for generating the analysis result is displayed according to an instruction from an examiner.

[0348] The display control unit 350 in the above-described embodiments and modifications can cause various diagnostic results, such as those related to glaucoma or age-related macular degeneration, to be displayed on a report screen of the display screen. At this time, for example, an accurate diagnostic result can be displayed by analyzing a medical image that has been processed for various artifact removals as described above. In addition, as a diagnostic result, the position of a specified abnormal part can be displayed on the image, and characters, etc. can be used to display the state of the abnormal part, etc. In addition, a classification result (e.g., Curtin's classification) for an abnormal part can be displayed as a diagnostic result. In addition, as a classification result, for example, information indicating the degree of possibility of each abnormal part (e.g., a numerical value indicating a percentage) can be displayed. In addition, information required so that a doctor can confirm the diagnosis can be displayed as a diagnostic result. For example, a suggestion such as performing additional imaging can be conceivable as the required information described above. For example, in the case where an abnormal part is detected in a vascular region in an OCTA image, information to the effect of suggesting that a doctor perform additional fluorescent imaging using a contrast agent that can observe blood vessels in more detail than OCTA can be displayed.

[0349] Note that the diagnosis result may be a result generated by the control unit 30, 1600, or 1900 using a learning model (a diagnosis result generation engine, or a learning model for diagnosis result generation) obtained by learning using the diagnosis result of a medical image as training data. In addition, the learning model may be a model obtained by learning using the following training data: training data including medical images and diagnosis results for the medical images, or training data including medical images and diagnosis results for medical images of a type different from that of related medical images, etc.

[0350] In addition, the training data may include a region label image generated by the segmentation process and the result of diagnosing the medical image using the region label image. In this case, the image processing units 320, 1620, and 1920 can be used as an example of a diagnosis result generating unit, for example, which generates a diagnosis result for the tomographic image from the result obtained by performing the segmentation process using the learning model for generating the diagnosis result (for example, the detection result of the retinal layer). In other words, by using a learning model (fifth learning model) for generating the diagnosis result that is different from the learning model for generating the high-quality image (the second medical image), the image processing units 320, 1620, and 1920 can generate a diagnosis result for each different region identified by the segmentation process.

[0351] In addition, a configuration may be adopted so as to display a diagnosis result obtained using a high-quality image generated using a learning model for improving image quality. In this case, the input data included in the training data may be a high-quality image generated using a learning model for improving image quality, or may be a set consisting of a low-quality image and a high-quality image. Note that the training data may be an image obtained by manually or automatically modifying at least a portion of an image whose image quality has been improved using a learning model.

[0352] In addition, the training data may be, for example, data obtained by marking (annotating) input data for which information including at least one of diagnosis, type or state (degree) of lesion (abnormal part), position of lesion in image, position of lesion relative to the region of interest, findings (explanation of findings, etc.), basis for diagnosis (affirmative medical support information, etc.), and basis for denial of diagnosis (negative medical support information), etc. is adopted as correct answer data (for supervised learning). Note that the following configuration may be adopted: a diagnosis result obtained using a learning model for diagnosis result generation is displayed according to an instruction from an examiner.

[0353] In addition, the display control unit 350 according to the various examples and modifications described above can cause the object recognition result (object detection result) or segmentation result for the focus, artifact, abnormal part, etc. as described above to be displayed on the report screen of the display screen. At this time, for example, a rectangular frame or the like can be superimposed around the object on the image and displayed. In addition, for example, a color or the like can be superimposed on the object on the image and displayed. Note that the object recognition result or the segmentation result can be a result generated using a learning model (object recognition engine, learning model for object recognition, segmentation engine, or learning model for segmentation) obtained by learning using training data, in which information indicating object recognition or segmentation is marked (annotated) as correct answer data on the medical image. Note that the above-mentioned analysis result generation or diagnosis result generation can be achieved by utilizing the above-mentioned object recognition result or segmentation result. For example, a process for generating an analysis result or for generating a diagnosis result can be performed for the focus obtained by the object recognition process or the segmentation process.

[0354] In addition, when an abnormal part is detected, the image processing units 320, 1620, and 1920 can use a generative adversarial network (GAN) or a variable autoencoder (VAE). For example, a DCGAN (Deep Convolutional GAN) composed of a generator obtained by learning to generate a tomographic image and a discriminator obtained by learning to distinguish between a new tomographic image generated by the generator and a true frontal image of the fundus can be used as a machine learning model.

[0355] In the case of using DCGAN, for example, the discriminator encodes the input tomographic image to convert the tomographic image into a latent variable, and the generator generates a new tomographic image based on the latent variable. Thereafter, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. In addition, in the case of using VAE, for example, the input tomographic image is converted into a latent variable by encoding the tomographic image using an encoder, and a new tomographic image is generated by decoding the latent variable using a decoder. Thereafter, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal part. Note that although the example of input data has been described using a tomographic image as an example, a fundus image or a frontal image of the anterior segment of the eye, etc. can also be used as input data.

[0356] In addition, the image processing units 320, 1620, and 1920 can detect abnormal parts using a convolutional autoencoder (CAE). In the case of using CAE, the same image is learned as input data and output data during learning. Therefore, when an image in which an abnormal part is present is input to CAE during estimation, an image in which no abnormal part is present is output according to the learning trend. Thereafter, the difference between the image input to CAE and the image output from CAE can be extracted as an abnormal part. Note that in this case as well, not only a tomographic image but also a fundus image or a frontal image of the anterior segment of the eye, etc. can be used as input data.

[0357] In these cases, the image processing units 320, 1620, and 1920 can generate information about the difference between the medical image obtained using the generative adversarial network or the autoencoder and the medical image input to the generative adversarial network or the autoencoder for each different region identified by the segmentation process, etc. as information about the abnormal part. Therefore, it can be expected that the image processing units 320, 1620, and 1920 will detect the abnormal part quickly and accurately. Here, examples of the autoencoder include VAE and CAE.

[0358] Furthermore, in the case of a diseased eye, the image features may differ depending on the type of disease. Therefore, a learning model used in the various examples and variations described above may be generated and prepared for each disease or each abnormal part. In this case, for example, the image processing unit 320 may select a learning model to be used for processing based on input (instructions) from the operator such as the type of disease or the abnormal part of the eye to be inspected. Note that the learning model prepared for each disease or each abnormal part is not limited to a learning model to be used for detecting the retinal layer or for generating a regional label image, etc., and may be, for example, a learning model to be used in an engine for evaluating images or in an engine for analysis, etc. At this time, the image processing units 320, 1620, and 1920 may use a separately prepared learning model to identify the type of disease or the abnormal part of the eye to be inspected from the image. In this case, the image processing units 320, 1620, and 1920 may automatically select a learning model to be used in the above-mentioned processing based on the type of disease or the abnormal part identified using a separately prepared learning model. Note that a learning model for identifying the type of disease or the abnormal part of the eye to be inspected can be learned using paired training data for which a tomographic image or fundus image, etc. is used as input data, and the type of disease or the abnormal part in these images is used as the ground truth. In this case, for the input data of the training data, the tomographic image or fundus image, etc. can be used independently as input data, or a combination of these images can be used as input data.

[0359] Moreover, in particular, the learning model for generating the diagnosis result may be a learning model obtained by learning using training data including input data, in which a plurality of medical images of different kinds as images of a predetermined part of the subject are taken as a set. At this time, for example, as input data included in the training data, it is conceivable to take a motion contrast front image and a luminance front image (or luminance tomographic image) of the fundus as data of the set. Furthermore, for example, as input data included in the training data, it is conceivable to take a tomographic image (B-scan image) and a color fundus image (or fluorescent fundus image) of the fundus as input data of the set. In addition, the plurality of medical images of different kinds may be of any kind as long as the medical images are obtained by different modalities, different optical systems, different principles, etc.

[0360] In addition, in particular, the learning model for generating the diagnosis result may be a learning model obtained by learning using training data including input data in which a plurality of medical images of different parts of the subject are taken as a set. At this time, for example, as input data included in the training data, it is conceivable to take a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment of the eye as input data of the set. In addition, for example, as input data included in the training data, it is also conceivable to take a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a tomographic image obtained by circular scanning (or raster scanning) of the optic nerve head of the fundus as input data of the set.

[0361] Note that the input data included in the training data may be a plurality of medical images of different parts and different kinds of the subject. At this time, for example, as the input data included in the training data, it is conceivable that a tomographic image of the anterior segment of the eye and a color fundus image are taken as a set of input data. In addition, the above-mentioned learning model may be a learning model obtained by learning using training data including input data, in which a plurality of medical images of different camera angles as images of a predetermined part of the subject are taken as a set. In addition, the input data included in the training data may be data obtained by joining together a plurality of medical images obtained by temporally dividing a predetermined part into a plurality of regions, for example, in the case of a panoramic image. At this time, by using a wide-angle image such as a panoramic image as training data, there is a possibility that the feature value of the image can be obtained with high accuracy due to reasons such as the fact that the amount of information is greater than that in the case of a narrow-angle image, and thus the result of the processing can be enhanced. For example, such a configuration is adopted so that, when an abnormal part is detected at a plurality of positions in a wide-angle image at the time of estimation (prediction), the enlarged images of the respective abnormal parts can be sequentially displayed. In this way, since, for example, abnormal parts at multiple positions can be effectively checked, the convenience of the examiner can be improved. For example, such a configuration can be adopted so that at this time, the examiner can select each position where the abnormal part is detected on the wide-angle image, and an enlarged image of the abnormal part at the selected position is displayed. In addition, the input data included in the training data can be input data in which a plurality of medical images obtained at different dates and times at a predetermined part of the subject are taken as a set.

[0362] In addition, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is to be displayed is not limited to the report screen. Such a display screen may be, for example, at least one of a camera confirmation screen, a display screen for tracking observation, and a preview screen for making various adjustments before camera shooting (a display screen on which various real-time motion images are displayed), etc. For example, by displaying at least one of the above-mentioned results obtained using the above-mentioned learning model on the camera confirmation screen, the examiner can check the accurate results immediately after the camera shooting. In addition, the display between the low-quality image and the high-quality image as described in Example 9, etc. may be, for example, the display between the analysis results of the low-quality image and the analysis results of the high-quality image.

[0363] The above-mentioned various learning models can be obtained by machine learning using training data. For example, deep learning composed of a multi-level neural network is a kind of machine learning. In addition, for example, a convolutional neural network (CNN) can be used as a machine learning model for at least a part of a multi-level neural network. In addition, a technique related to an autoencoder can be used for at least a part of a multi-level neural network. In addition, a technique related to back propagation (error back propagation method) can be used for learning. However, this machine learning is not limited to deep learning, and any learning can be used as long as the learning uses a model that can extract (represent) the feature values ​​of training data such as images by learning. Here, the term "machine learning model" refers to a learning model according to a machine learning algorithm such as deep learning. In addition, the term "learning model" refers to a model that is trained (learned) in advance using appropriate training data for a machine learning model according to any machine learning algorithm. However, it is assumed that the learning model is not a model that does not perform further learning, and is a model that can also perform incremental learning. In addition, the term "training data" refers to data composed of paired input data and output data (correct answer data). Here, in some cases, training data is also referred to as "teaching data", and in some cases, correct answer data is also referred to as "teaching data".

[0364] Note that the GPU can perform efficient arithmetic operations by performing parallel processing of a large amount of data. Therefore, it is effective to use the GPU for processing in the case of multiple learning using a learning model such as deep learning. Therefore, in this modification, in addition to the CPU, the GPU is also used for processing of the image processing units 320, 1620 and 1920 as examples of the learning unit (not shown). Specifically, when a learning program including a learning model is executed, learning is performed by the CPU and GPU that cooperate to perform arithmetic operations. Note that for the processing of the learning unit, arithmetic operations can be performed only by the CPU or the GPU. In addition, similar to the learning unit, the processing unit (estimation unit) that performs processing using the various learning models described above can also use the GPU. The learning unit may also include an error detection unit and an update unit (not shown). The error detection unit obtains the error between the output data output from the output layer of the neural network according to the input data input to the input layer and the correct answer data. The error detection unit may be constructed to calculate the error between the output data from the neural network and the correct answer data using a loss function. In addition, based on the error obtained by the error detection unit, the update unit updates the combined weighting factors between the nodes of the neural network, etc., thereby making the error smaller. The updating unit updates the combination weighting factors, etc., using, for example, an error back propagation method. The error back propagation method is a method of adjusting the combination weighting factors, etc. between each neural network node so as to reduce the above-mentioned error.

[0365] In addition, a U-Net type machine learning model can be applied as a machine learning model for image quality improvement or segmentation, etc., the U-Net type machine learning model has the function of an encoder and the function of a decoder, the encoder is composed of multiple layers, the multiple layers include multiple downsampling layers, and the decoder is composed of multiple layers, the multiple layers include multiple upsampling layers. In the U-Net type machine learning model, position information (spatial information) that has become blurred in the multiple layers constructed as an encoder is constructed (for example, using skip connections) so that the information can be used in layers of the same dimension (layers corresponding to each other) in the multiple layers constructed as a decoder.

[0366] In addition, for example, FCN (Fully Convolutional Network) or SegNet, etc. can also be used as a machine learning model for image quality improvement or segmentation, etc. In addition, a machine learning model for object recognition in units of regions can be used according to the desired structure. As a machine learning model for object recognition, for example, RCNN (Regional CNN), Fast-RCNN, or Faster-RCNN can be used. In addition, YOLO (Look Only Once) or SSD (Single Shot Detector or Single Shot MultiBox Detector) can also be used as a machine learning model for object recognition in units of regions.

[0367] In addition, the machine learning model may be, for example, a capsule network (CapsNet). In this case, in a common neural network, by constructing each unit (each neuron) to output a scalar value, the neural network is constructed so that, for example, spatial information related to the spatial position relationship (relative position) between features in an image is reduced. In this way, for example, learning can be performed in which the influence of local distortion or parallel displacement in the image is reduced. On the other hand, in a capsule network, each unit (each capsule) is constructed to output spatial information as a vector, and, for example, is constructed to maintain spatial information. In this way, for example, learning can be performed in which the spatial position relationship (relative position) between features in an image is taken into account.

[0368] In addition, the image quality improvement model (learning model for improving image quality) may be a learning model obtained by incremental learning using training data including at least one high-quality image generated by the image quality improvement model. At this time, the following construction may be adopted: it is possible to make a selection as to whether to use the high-quality image as training data for incremental learning to be performed according to the instruction from the examiner. Please note that these constructions are not limited to the learning model for improving image quality, and may also be applied to the various learning models described above. In addition, the learning model for generating correct answer data such as marking (annotation) may be used to generate the correct answer data used for learning the various learning models described above. At this time, the learning model for generating the correct answer data may be a learning model obtained by performing (sequential) incremental learning of the correct answer data obtained when the examiner performs marking (annotation). In other words, the learning model for generating the correct answer data may be a learning model obtained by performing incremental learning on the training data, in which the data before marking is used as input data, and the data after marking is used as output data. In addition, in the case of multiple consecutive frames such as a moving image, such a construction may also be adopted: considering the results of object recognition or the segmentation of the previous frame and the next frame, the result is modified for the frame with low accuracy of the determination result. At this time, a configuration may be adopted in which the modified result is used as correct answer data for incremental learning according to an instruction from the examiner.

[0369] Note that in the various embodiments and modifications described above, in the case of detecting an area of ​​the eye to be inspected using a learning model for object recognition or a learning model for segmentation, predetermined image processing may also be performed on each detected area. For example, let us consider a case where at least two areas of the vitreous body area, the retinal area, and the choroidal area are detected. In this case, when image processing such as contrast adjustment is performed on at least two detected areas, adjustments suitable for each area may be performed by using different image processing parameters for each area. By displaying an image to which adjustments suitable for each area have been made, an operator may more appropriately diagnose diseases in each area, etc. Note that for a construction using different image processing parameters for each detected area, for example, such a construction may also be similarly applied to an area of ​​the eye to be inspected that is detected without using a learning model.

[0370] (Variant 11)

[0371] The following structure can be adopted: on the preview screen in the various examples and modifications described above, the above-mentioned learning model for improving image quality is used for each at least one frame of the real-time motion image. In addition, the following structure can also be configured: when a plurality of real-time motion images of different parts or different types are displayed on the preview screen, the learning model corresponding to each real-time motion image is used. In this way, for example, since the processing time can be shortened even for real-time motion images, the examiner can obtain high-precision information before starting imaging. Therefore, for example, since troubles such as re-imaging can be reduced, the accuracy and efficiency of diagnosis can be improved.

[0372] Note that the multiple real-time motion images may include, for example, motion images of the anterior segment of the eye for alignment in the XYZ direction, and frontal motion images of the fundus for focusing of the OCT or focusing of the fundus observation optical system. In addition, the multiple real-time motion images may also include, for example, tomographic motion images of the fundus for coherence gate adjustment in OCT (adjustment of the optical path length difference between the measured optical path length and the reference optical path length), etc. At this time, the following construction may be adopted: the above-mentioned various adjustments are made so that the area detected using the learning model for object recognition or the learning model for segmentation as described above satisfies the predetermined conditions. For example, the following construction may be adopted: various adjustments such as OCT focus adjustment are made so that the value (e.g., contrast value or brightness value) related to the vitreous area or a predetermined retinal layer such as RPE detected using the learning model for object recognition or the learning model for segmentation exceeds a threshold value (or becomes a peak value). Furthermore, for example, a configuration may be adopted in which coherence gate adjustment in OCT is performed so that a vitreous region or a predetermined retinal layer such as RPE detected using a learning model for object recognition or a learning model for segmentation is at a predetermined position in the depth direction.

[0373] In these cases, the image quality improvement unit 322 or 1622 can perform image quality improvement processing for the moving image using the learning model, thereby generating a high-quality moving image. In addition, in a state in which a high-quality moving image is displayed, the drive control unit 330 can perform drive control of an optical member that changes the imaging range of the reference mirror 221 or the like, so that any one of the different areas identified by the segmentation processing or the like is located at a predetermined position in the display area. In this case, the control unit 30, 1600 or 1900 can automatically perform alignment processing based on high-precision information so that the desired area is located at a predetermined position in the display area. Note that the optical member that changes the imaging range may be, for example, an optical member that adjusts the coherence gate position, and specifically may be the reference mirror 221 or the like. In addition, the coherence gate position may be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member in question may be, for example, a mirror (not shown) for changing the optical path length of the measurement light or the like. Note that the optical member that changes the imaging range may also be, for example, the stage unit 25.

[0374] In addition, the motion images to which the above-mentioned learning model can be applied are not limited to real-time motion images, and, for example, the motion images can be motion images stored (saved) in a memory. At this time, for example, a motion image obtained by aligning at least one frame of a fundus tomography motion image stored (saved) in a memory can be displayed on a display screen. For example, in the case where it is desired to properly observe the vitreous region, first, a reference frame can be selected based on conditions such as the vitreous region being present in the frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Subsequently, a motion image that has been aligned with other frames in the XZ direction for the selected reference frame can be displayed on the display screen. At this time, for example, a structure can be adopted so that high-quality images (high-quality image frames) generated sequentially for at least one frame of a motion image by a learning model for improving image quality are continuously displayed.

[0375] Note that as the above-mentioned method for aligning between frames, the same method may be applied to the method for aligning in the X direction and the method for aligning in the Z direction (depth direction), or the methods applied may be different. In addition, alignment in the same direction may be performed multiple times by different methods. For example, coarse alignment may be performed, and thereafter fine alignment may be performed. In addition, as an alignment method, for example, a method for performing (coarse Z direction) alignment using the retinal layer boundary obtained by segmenting the tomographic image (B-scan image) may be used. In addition, as an alignment method, for example, a method for performing (fine X direction or Z direction) alignment using the related information (similarity) between a plurality of regions obtained by dividing the tomographic image and the reference image may also be used. As other methods for alignment, for example, a method for performing (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image) may be used, and a method for performing (X direction) alignment using a two-dimensional frontal image generated for each tomographic image (B-scan image) or the like may be used. In addition, a configuration may be adopted so that fine alignment is performed in sub-pixel units after coarse alignment is performed in pixel units.

[0376] In this case, there is a possibility that the imaging target such as the retina of the subject's eye cannot be successfully imaged during various adjustments. Therefore, since there is a large difference between the medical image input to the learning model and the medical image used as training data, there is a possibility that a high-quality image cannot be accurately obtained. Therefore, the following structure can also be adopted: when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B-scan) exceeds a threshold, the display of high-quality motion images (continuous display of high-image quality frames) is automatically started. In addition, the following structure can be adopted: when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B-scan) exceeds a threshold, the image quality improvement button is changed to a state (activated state) in which the button can be specified by the examiner.

[0377] In addition, the following configuration may be adopted: different learning models for improving image quality are prepared for different imaging modes such as scanning patterns, and a learning model for improving image quality corresponding to the selected imaging mode is selected. In addition, a learning model for improving image quality obtained by learning using training data including various medical images obtained in different imaging modes may be used.

[0378] (Variant 12)

[0379] In the various examples and variations described above, in the case where various learning models are undergoing incremental learning, there is a possibility that it will be difficult to output (infer / predict) using the learning model undergoing incremental learning itself. Therefore, it is possible to prohibit the input of medical images into the learning model undergoing incremental learning. In addition, a learning model that is the same as the learning model undergoing incremental learning is prepared as another auxiliary learning model. At this time, the following construction can also be adopted: so that the input of medical images to the auxiliary learning model can be performed while incremental learning is being performed. Subsequently, after the incremental learning is completed, the learning model that has undergone incremental learning is evaluated, and if there is no problem, it is sufficient to switch from the auxiliary learning model to the learning model that has undergone incremental learning. In addition, the following construction can be adopted: if there is a problem, the auxiliary learning model is used.

[0380] In addition, the following configuration may be adopted: so that a learning model obtained by learning for each imaging part can be selectively utilized. Specifically, a plurality of learning models may be prepared, including a first learning model obtained using training data including a first imaging part (lung, eye to be inspected, etc.), and a second learning model obtained using training data including a second imaging part, which is different from the first imaging part. In addition, the image processing units 320, 1620, and 1920 may have a selection unit for selecting any one of the plurality of learning models. At this time, the image processing units 320, 1620, and 1920 may have a control unit for performing incremental learning for the selected learning model. The control unit may retrieve data in which the imaging part corresponding to the selected learning model and the image obtained by imaging the relevant imaging part are paired according to an instruction from the examiner, and perform learning in which the retrieved and obtained data is used as training data as incremental learning for specifying the selected learning model. Note that the imaging part corresponding to the selected learning model may be a part obtained based on the title information of the data, or a part manually input by the examiner. In addition, data can be retrieved from a server of an external facility such as a hospital or laboratory, for example, via a network. In this way, by using images obtained by imaging an imaging part corresponding to a learning model, incremental learning can be efficiently performed for each imaging part.

[0381] Note that the selection unit and the control unit may be constituted by software modules executed by a processor such as an MPU or a CPU of the control unit 30, 1600 or 1900. In addition, the selection unit and the control unit may be constituted by a circuit having a specific function such as an ASIC or an independent device.

[0382] Furthermore, when training data for incremental learning is obtained from a server of an external facility such as a hospital or a laboratory via a network, it is useful to reduce the reduction in reliability due to tampering or system failure during incremental learning, etc. Therefore, the correctness of the training data for incremental learning can be detected by confirming consistency through a digital signature or hash. In this way, the training data for incremental learning can be protected. At this time, in the event that the correctness of the training data for incremental learning cannot be detected as a result of confirming consistency through a digital signature or hash, a warning is given for this, and the training data in question is not used for incremental learning. Note that the server may be any form of server such as a cloud server, a fog server, or an edge server, regardless of its installation location.

[0383] (Variant 13)

[0384] In the various example variations described above, in addition to manual instructions (for example, instructions using a user interface, etc.), the instructions from the inspector may also be voice instructions, etc. At this time, for example, a machine learning model including a voice recognition model obtained through machine learning (a voice recognition engine or a learning model for voice recognition) may be used. In addition, the manual instructions may be instructions input through characters using a keyboard, a touch panel, etc. At this time, for example, a machine learning model including a character recognition model obtained through machine learning (a character recognition engine or a learning model for character recognition) may be used. In addition, the instructions from the inspector may be instructions made through gestures, etc. At this time, a machine learning model including a gesture recognition model obtained through machine learning (a gesture recognition engine or a learning model for gesture recognition) may be used.

[0385] In addition, the instruction from the examiner may be a result of detecting the line of sight of the examiner on the display screen of the display unit 50. The line of sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from around the display screen of the display unit 50. At this time, pupil detection from the moving image may use the object recognition engine as described above. In addition, the instruction from the examiner may be an instruction of a brain wave, or a weak electrical signal flowing through the body, etc.

[0386] In this case, for example, the training data may be the following training data: character data or voice data (waveform data) or the like indicating instructions for displaying the results obtained by processing the above-mentioned various learning models is used as input data, and an execution command for causing the results obtained by processing the various learning models to be actually displayed on the display unit is used as correct answer data. In addition, the training data may be the following training data: for example, character data or voice data or the like indicating instructions for displaying a high-quality image obtained by a learning model for improving image quality is used as input data, and an execution command for displaying a high-quality image and an execution command for causing the results obtained by processing the various learning models to be actually displayed on the display unit are used as correct answer data. Fig.22A and Fig. 22B The button 2220 shown changes to an execution command in an activated state as the correct answer data. Please note that any kind of training data can also be used, as long as the instruction content and the execution command content indicated by the character data or voice data, etc. correspond to each other. In addition, the voice data can be converted into character data using an acoustic model or a language model, etc. In addition, the waveform data obtained using multiple microphones can be used to reduce the processing of noise data superimposed on the voice data. In addition, the following structure can be adopted: it is possible to select between instructions issued by characters or voices, etc. and instruction input using a mouse or a touch panel, etc., according to instructions from the inspector. In addition, the following structure can be adopted: it is possible to select to turn on or off instructions through characters or voices, etc., according to instructions from the inspector.

[0387] In this case, the machine learning includes deep learning as described above, and, for example, a recurrent neural network (RNN) may be used as at least a part of a multilayer neural network. Here, as an example of a machine learning model according to this variant, reference will be made to 24A and Fig. 24B The RNN as a neural network for processing time series information is described. In addition, reference will be made to 25A and Fig.25B A long short-term memory (hereinafter referred to as “LSTM”) as a type of RNN is described.

[0388] Fig.24A The structure of RNN as a machine learning model is shown. RNN 2420 has a loop structure in the network, and the data x t 2410 is input to RNN 2420 at time t, and RNN 2420 outputs data h t 2430. Since the RNN 2420 has a loop function in the network, the state of the current time can be taken over to the next state, so the timing information can be processed. Fig. 24B An example of the input / output of a parameter vector at time t is shown. Data x t 2410 includes N data (parameter 1 to parameter N). In addition, the data h output by RNN 2430 t 2420 includes N pieces of data (parameter 1 to parameter N) corresponding to the input data.

[0389] However, since RNN cannot process long-term information during back-propagation, LSTM can be used. LSTM can learn long-term information by providing a forget gate, input gate, and output gate. Fig.25A The structure of LSTM is shown. In LSTM 2540, the information that the network takes over at the next time t is the internal state c of the network called the cell. t -1, and output data h t -1. Note that lowercase letters (c, h, x) in the diagram represent vectors.

[0390] Next, in Fig.25B LSTM 2540 is shown in detail in FIG. The forget gate network FG, the input gate network IG and the output gate network OG are shown in FIG. Fig.25B As shown in , each of these networks is an S-shaped layer. Therefore, a vector in which each element has a value from 0 to 1 is output. The forget gate network FG determines how much past information is retained, and the input gate network IG determines which value will be updated. Fig.25BThe cell update candidate network CU is also shown in FIG, and the cell update candidate network CU is an activation function tanh layer. This will create a vector of new candidate values ​​that will be added to the cell. The output gate network OG selects the elements of the cell candidate and selects how much information to transmit at the next time.

[0391] Please note that the above LSTM model is a basic form, and the present invention is not limited to the network shown here. The coupling between the networks can be changed. QRNN (quasi-recurrent neural network) can be used instead of LSTM. In addition, the machine learning model is not limited to neural networks, and Boosting or Support Vector Machine (Support Vector Machine) can be used. In addition, in the case where instructions from the inspector are input by characters or voice, etc., techniques related to natural language processing (e.g., sequence to sequence) can be applied. In addition, a dialogue engine (dialogue model or learning model for dialogue) that responds to the inspector with output such as text or voice can be applied.

[0392] (Variant 14)

[0393] In the various examples and variations described above, a high-quality image, a label image, and the like can be stored in a memory according to an instruction from an operator. At this time, for example, after an instruction to save a high-quality image from an operator, when registering a file name, a file name including information (e.g., characters) indicating that the image is an image generated by processing (image quality improvement processing) using a learning model for improving image quality at any part of the file name (e.g., the first part or the last part) can be displayed as a recommended file name in a state in which the file name can be edited according to an instruction from an operator. Note that for a boundary image or a region label image, etc., a file name including information indicating that the image is an image generated by processing using a learning model can also be displayed in a similar manner.

[0394] Furthermore, when the display unit 50 is caused to display a high-quality image on various display screens such as a report screen, a display indicating that the image being displayed is a high-quality image generated by processing using a learning model for improving image quality may be displayed together with the high-quality image. In this case, since the operator can easily recognize through the relevant display that the displayed high-quality image is not an actual image obtained by photographing, misdiagnosis can be reduced, and diagnostic efficiency can be improved. Note that the display indicating that the high-quality image is generated by processing using a learning model for improving image quality may be in any form as long as it is a display that makes it possible to distinguish between an input image and a high-quality image generated by the relevant processing. Furthermore, also for processing using various learning models described above, not just processing using a learning model for improving image quality, a display indicating that the result being displayed is generated by processing using a learning model of a relevant kind may be displayed together with the relevant result. Furthermore, when an analysis result regarding a segmentation result obtained using a learning model for segmentation processing is also displayed, a display indicating that the analysis result is a result obtained based on the use of a learning model for segmentation may be displayed together with the analysis result.

[0395] At this time, a display screen such as a report screen can be stored in the memory as image data according to an instruction from the operator. For example, the report screen can be stored in the memory as a single image in which high-quality images and the like and a display indicating that these images are generated by processing using a learning model are displayed side by side.

[0396] Furthermore, with respect to a display indicating that a high-quality image is generated by processing using a learning model for improving image quality, a display indicating what kind of training data is used by the learning model for improving image quality when learning is performed may be displayed on the display unit. The display in question may include a display of a description of the types of input data and correct answer data of the training data, or any display related to the input data and the correct answer data such as the imaging part included in the correct answer data. Note that with respect to processing such as segmentation processing using various learning models as described above, a display indicating what kind of training data is used by the learning model of the relevant type when learning is performed may also be displayed on the display unit.

[0397] The following configuration may also be adopted: information indicating that the image is generated by processing using a learning model (e.g., characters) is displayed or stored in a state where the information is superimposed on an image or the like. At this time, the position where the information is superimposed on the image may be any position as long as the position is in an area (e.g., the edge of the image) that does not overlap with an area where a part of interest or the like that is a photographic target is displayed. Furthermore, a non-superimposed area may be determined, and the information may be superimposed in the determined area. Note that, for example, an image obtained by processing using the various learning models described above, such as segmentation processing, may be processed in a similar manner, not just for processing using a learning model for improving image quality.

[0398] Furthermore, a configuration may be adopted such that, in the initial display screen as the report screen, the default setting is set such that Fig.22A and Fig. 22B When the button 2220 shown enters an activated state (the image quality improvement process is set to "on"), a report image corresponding to a report screen including a high-quality image, etc. is sent to the server in accordance with an instruction from the examiner. In addition, the following configuration may be adopted: in a case where the default setting is set so that the button 2220 enters an activated state, when the inspection is finished (for example, in a case where the image confirmation screen or the preview screen is changed to the report screen in accordance with an instruction from the examiner), a report image corresponding to the report screen including a high-quality image, etc. is (automatically) sent to the server. At this time, the following configuration may be adopted: a report image generated based on various settings of the default setting (for example, settings related to at least one of the depth range for generating an en-face image on the initial display screen of the report screen, whether to superimpose an analysis diagram, whether the image is a high-quality image, and whether to display a display screen for tracking observation, etc.) is sent to the server. Note that similar processing can also be performed in the case where the button 2220 indicates switching of the segmentation process.

[0399] (Variant 15)

[0400] In the various examples and variations described above, in the various learning models described above, an image obtained using a first learning model (e.g., a high-quality image, an image showing an analysis result such as an analysis graph, an image showing an object recognition result, or an image showing a segmentation result) can be input into a second learning model that is different from the first learning model. At this time, the following construction can be adopted: a result (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result) is generated by processing the second learning model.

[0401] In addition, in the various learning models described above, an image to be input into a second learning model different from the first learning model can be generated from an image input into the first learning model by using a result obtained by processing the first learning model (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result). At this time, the generated image is likely to be an image suitable as an image for processing the second learning model. Therefore, the accuracy of the image obtained when the generated image is input into the second learning model (e.g., a high-quality image, an image showing an analysis result such as an analysis graph, an image showing an object recognition result, or an image showing a segmentation result) is enhanced.

[0402] In addition, similar case images can be retrieved using an external database stored in a server or the like, using the analysis results or diagnosis results obtained by processing the above-mentioned learning model as a search keyword. Note that in the case where multiple images stored in a database have been managed in a state where the individual feature values ​​of the multiple images have been attached as supplementary information by machine learning or the like, a similar case image search engine (similar case image search model or learning model for similar case image search) using the image itself as a search keyword can be used. For example, the image processing units 320, 1620, and 1920 can search for similar case images for each different region identified by segmentation processing or the like by using a learning model (sixth learning model) for similar case image search that is different from the learning model used to generate the high-quality image (second medical image).

[0403] (Variant 16)

[0404] Note that the processing for generating motion contrast data in the above-described examples and modifications is not limited to the configuration in which processing is performed based on the luminance value of a tomographic image. The above-described various processing can be applied to interference signals obtained with the OCT imaging device 20, signals obtained by Fourier transforming interference signals, signals obtained by performing any processing on correlation signals, and tomographic data including tomographic images and the like based on these signals. In these cases as well, effects similar to those of the aforementioned configuration can be obtained.

[0405] Furthermore, the image processing such as the grayscale conversion processing in the above-described embodiment and the modified example is not limited to the configuration in which the processing is performed based on the luminance value of the tomographic image. The above-described various processing can be applied to tomographic data including the interference signal obtained with the imaging unit 20, the signal obtained by performing Fourier transform on the interference signal, and the signal obtained by performing any processing on the correlation signal. In these cases as well, effects similar to those of the aforementioned configuration can be obtained.

[0406] In addition, although a spectral domain OCT (SD-OCT) device using an SLD as a light source is described as an OCT device in the above examples and modifications, the construction of the OCT device according to the present invention is not limited thereto. For example, the present invention may also be applied to a swept source OCT (SS-OCT) device using a wavelength swept light source capable of scanning the wavelength of emitted light or any other type of OCT device. Furthermore, the present invention may also be applied to a Line-OCT device (or SS-Line-OCT) using line light. Furthermore, the present invention may also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light.

[0407] In addition, although the optical fiber system using a coupler as a separation unit is used in the above examples and modifications, a spatial optical system of a collimator and a beam splitter may also be used. In addition, the configuration of the imaging unit 20 is not limited to the above configuration, and some components included in the imaging unit 20 may be provided as components separated from the imaging unit 20.

[0408] Furthermore, in the above-described examples and modifications, the obtaining unit 310 obtains the interference signal obtained by the imaging unit 20, the tomographic image generated by the image processing unit 320, and the like. However, the configuration in which the obtaining unit 310 obtains these signals or images is not limited to the above configuration. For example, the obtaining unit 310 may obtain these signals from a server or an imaging device connected to the control unit 30, 1600, or 1900 via a LAN, WAN, or the Internet, or the like.

[0409] In addition, the learning data of various learning models is not limited to data obtained using the ophthalmic device itself that performs actual imaging, and depending on the desired construction, the learning data may be data obtained using an ophthalmic device of the same model, or may be data obtained using the same type of ophthalmic device.

[0410] Note that various learning models according to the above-described embodiments and variations may be set in the control units 30, 1600, and 1900. For example, these learning models may be composed of software modules executed by processors such as a CPU, an MPU, a GPU, or an FPGA, or may be composed of circuits having specific functions such as an ASIC. In addition, these learning models may be set in different devices such as servers connected to the control unit 30, 1600, or 1900. In this case, the control unit 30, 1600, or 1900 may be connected to a server including a learning model, etc., through any network such as the Internet to use the learning model. The server including the learning model may be, for example, a cloud server, a FOG server, or an edge server.

[0411] Please note that although the tomographic images related to the fundus portion of the eye to be inspected are described in the above embodiments and modifications, the above-mentioned image processing can also be performed on the tomographic images related to the anterior segment of the eye to be inspected. In this case, the areas to be subjected to different image processing in the tomographic images include areas such as the lens, cornea, iris, and anterior chamber. Please note that the areas in question may also include other areas of the anterior segment of the eye. In addition, the areas for the tomographic images related to the fundus portion are not limited to the vitreous body, the retina, and the choroid, and may also include other areas related to the fundus portion. In this regard, since the grayscale range of the tomographic images related to the fundus portion is wider than the grayscale range of the tomographic images related to the anterior segment of the eye, image quality improvement can be more effectively performed by image processing according to the above-mentioned embodiments and modifications.

[0412] In addition, although the eye to be inspected is described as an example of the object to be inspected in the above embodiments and modifications, the object to be inspected is not limited thereto. For example, the object to be inspected may be skin or another organ. In this case, the OCT device according to the above embodiments and modifications may be applied to medical devices other than ophthalmic devices, such as endoscopes.

[0413] (Variant 17)

[0414] In addition, the images to be processed by the image processing device or the image processing method according to the above-mentioned various examples and variations include medical images obtained using any modality (camera or imaging method). The medical images to be processed may include medical images obtained by any imaging device, etc., as well as images generated by the image processing device or the image processing method according to the above-mentioned examples and variations.

[0415] In addition, the medical image to be processed is an image of a predetermined part of the subject (examinee), and the image of the predetermined part includes at least a part of the predetermined part of the subject. The medical image may also include another part of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. In addition, the medical image may be an image representing the structure (form) of the predetermined part, or may be an image representing the function of the predetermined part. For example, images representing functions include images representing hemodynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. Note that the predetermined part of the subject can be determined based on the imaging target, and these predetermined parts include any parts, for example, organs such as the human eye (eye to be examined), brain, lungs, intestines, heart, pancreas, kidneys, and liver, as well as the head, chest, legs, and arms.

[0416] In addition, the medical image may be a tomographic image of the subject, or may be a frontal image. Examples of frontal images include a frontal image of the fundus, a frontal image of the anterior ocular segment, a fundus image obtained by fluorescence imaging, and an en-face image generated using data of at least a partial range in the depth direction of the imaging target for data obtained by OCT (three-dimensional OCT data). Note that the en-face image may be an OCTA en-face image (motion contrast frontal image) generated using data of at least a partial range in the depth direction of the imaging target for three-dimensional OCTA data (three-dimensional motion contrast data). In addition, three-dimensional OCT data or three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0417] Here, the term "motion contrast data" refers to data showing changes between a plurality of volume data items obtained by controlling so that the measuring light is scanned multiple times on the same area (same position) of the eye to be inspected. At this time, the volume data is composed of a plurality of tomographic images obtained at different positions. Then, at each position different from each other, by obtaining data showing changes between a plurality of tomographic images obtained at approximately the same position, the motion contrast data can be obtained as volume data. Note that with respect to OCT angiography (OCTA) for measuring blood flow motion, the motion contrast en-face image is also referred to as an OCTA en-face image, and the motion contrast data is also referred to as OCTA data. The motion contrast data can be obtained, for example, as a variance value or decorrelation value between two tomographic images or between interference signals corresponding to the two tomographic images, or as a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value), and can be obtained by any known method. At this time, two tomographic images can be obtained, for example, by controlling so that the measuring light is scanned multiple times on the same area (same position) of the eye to be inspected.

[0418] In addition, the en-face image is a frontal image generated by, for example, projecting data of a range between two layer boundaries in the X direction and the Y direction. At this time, the frontal image is generated by projecting or integrating data corresponding to a depth range onto a two-dimensional plane, the depth range being at least a local depth range of volume data (three-dimensional tomographic image) obtained using optical interference and defined based on two reference planes. The en-face image is a frontal image generated by projecting data corresponding to a depth range onto a two-dimensional plane, the depth range being determined based on the detected retinal layer in the volume data. Note that as a technique for projecting data corresponding to a depth range defined based on two reference planes on a two-dimensional plane, for example, a technique for using a representative value of data within a relative depth range as a pixel value on a two-dimensional plane can be used. In this case, the representative value may include values ​​such as an average value, a median value, or a maximum value of pixel values ​​within a range in the depth direction of an area surrounded by two reference planes. In addition, the depth range associated with the en-face image may be a range that includes only a predetermined number of pixels in a deeper direction or a shallower direction for one of the two layer boundaries associated with the detected retinal layer. In addition, the depth range related to the en-face image may be a range that has been changed (shifted) from the range between two layer boundaries related to the detected retinal layers according to an instruction of an operator, for example.

[0419] In addition, the term "imaging device" refers to a device for performing imaging to obtain an image to be used for diagnosis. Examples of imaging devices include a device for obtaining an image of a predetermined portion of a subject by irradiating the predetermined portion with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, and a device for obtaining an image of a predetermined portion by detecting radiation emitted from the subject. More specifically, examples of imaging devices according to the various examples and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

[0420] Note that a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device may be included as examples of an OCT device. Furthermore, examples of a Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a swept source OCT (SS-OCT) device. Furthermore, an adaptive optical SLO (AO-SLO) device and an adaptive optical OCT (AO-OCT) device using an adaptive optical system or the like may be included as examples of an SLO device or an OCT device, respectively. Furthermore, a polarization-sensitive SLO (PS-SLO) device and a polarization-sensitive OCT (PS-OCT) device for visualizing information related to a polarization phase difference or depolarization may be included as examples of an SLO device or an OCT device, respectively.

[0421] Furthermore, in the learning model for improving image quality according to the various examples and variations described above, the magnitude of the luminance value of the tomographic image, and the order and slope, position, distribution, and continuity of light and dark parts of the tomographic image to be extracted as part of the feature value and used for the estimation process, etc. can be thought of. Similarly, in the case of a learning model also used for segmentation processing, for image analysis, and for generating a diagnosis result, the magnitude of the luminance value of the tomographic image, and the order and slope, position, distribution, and continuity of light and dark parts of the tomographic image to be extracted as part of the feature value and used for the estimation process, etc. can be thought of. On the other hand, in the case of a learning model for speech recognition, for character recognition, for gesture recognition, etc., since learning using time series data is performed, it can be thought of extracting the slope between the input continuous time series data values ​​as part of the feature value, and using the slope for the estimation process. Therefore, it is expected that by utilizing the influence of the change over time of the specific numerical value in the estimation process, it is possible to use such a learning model for good accuracy estimation.

[0422] According to the above-described embodiment and the modified examples, it is possible to generate an image that appears to be an image on which appropriate image processing has been performed on each region as an observation target.

[0423] (Other examples)

[0424] The present invention can also be implemented by processing in the following manner: a program that implements one or more functions according to the above-mentioned examples and variants is supplied to a system or device via a network or a storage medium, and a computer of the system or device reads the program and executes it. The computer has one or more processors or circuits, and in order to read and execute computer-executable instructions, the computer may include a plurality of computers separated from each other, a plurality of processors separated from each other, or a circuit network.

[0425] Examples of processors or circuits may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). In addition, examples of processors or circuits may include a digital signal processor (DSP), a data stream processor (DFP), or a neural processing unit (NPU).

[0426] The present invention is not limited to the above-described embodiments and variations, and various changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are attached.

[0427] This application claims the benefit of priority from Japanese Patent Application No. 2018-191449, filed on October 10, 2018, and Japanese Patent Application No. 2019-183106, filed on October 3, 2019, which are hereby incorporated by reference herein in their entirety.

[0428] [reference numerals list]

[0429] 30 control unit (image processing device);

[0430] 310 Get Unit;

[0431] 322 Image quality improvement unit.

Claims

1. An image processing device, comprising: an obtaining unit configured to obtain a first tomographic image of the eye to be inspected obtained by using optical coherence; as well as An image quality improving unit, configured to generate a second tomographic image having a plurality of regions, the plurality of regions including a retinal region and a vitreous region having a contrast higher than that of the plurality of regions in the first tomographic image obtained, using the obtained first tomographic image as input data input into a learning model, the learning model being obtained by using training data, the training data including a third tomographic image and a fourth tomographic image of an eye to be inspected obtained by using optical coherence, the third tomographic image having at least one region among the plurality of regions including the retinal region and the vitreous region, the fourth tomographic image having at least one region among the plurality of regions having a contrast higher than that of at least one region among the plurality of regions in the third tomographic image, and the training data including a fifth tomographic image and a sixth tomographic image of the eye to be inspected obtained by using optical coherence, the fifth tomographic image having at least another region among the plurality of regions, and the sixth tomographic image having at least another region among the plurality of regions having a contrast higher than that of at least another region among the plurality of regions in the fifth tomographic image.

2. The image processing device according to claim 1, further comprising: a selection unit configured to select image processing to be applied to the first tomographic image according to an instruction from an operator; Here, based on the image processing selected by the selection unit, the image quality improvement unit performs grayscale conversion processing on the first tomographic image without using the learning model to generate a seventh tomographic image, or generates a second tomographic image from the first tomographic image using the learning model.

3. The image processing apparatus according to claim 1, wherein: In the second tomographic image, the image quality improving unit modifies the pixel value of the connection between the plurality of regions different from each other in the first tomographic image based on the pixel values ​​of pixels around the connection, or modifies the pixel values ​​of surrounding pixels based on the pixel value of the connection.

4. The image processing apparatus according to claim 1, wherein: The plurality of regions also include a choroidal region.

5. The image processing apparatus according to claim 1, wherein: The training data of the learning model includes a tomographic image obtained by subjecting a tomographic image obtained in an imaging mode corresponding to a retinal area in a tomographic image obtained by imaging the eye to be examined to grayscale conversion processing corresponding to a vitreous area in the tomographic image obtained by imaging the eye to be examined.

6. The image processing apparatus according to claim 1, wherein: The training data of the learning model includes a tomographic image obtained by subjecting the tomographic image obtained by using optical coherence to image processing for improving contrast of each region; and The image processing for improving the contrast is one of averaging processing, maximum a posteriori processing, smoothing filtering processing, and grayscale conversion processing.

7. The image processing device according to claim 1, further comprising: an analysis unit configured to apply different analysis conditions to each of the plurality of regions in the second tomographic image; as well as A display control unit is configured to cause the second tomographic image and an analysis result obtained by the analysis unit with respect to each of the plurality of regions to be displayed on a display unit.

8. The image processing device according to claim 1, further comprising: a display control unit configured to control display of the display unit; Here, the display control unit causes information indicating that the second tomographic image is an image generated using the learning model to be displayed on the display unit together with the second tomographic image.

9. The image processing apparatus according to claim 1, wherein: The image quality improvement unit: generating, from the first tomographic image, a label image in which different label values ​​are assigned to a plurality of regions by using a learning model different from a learning model used to generate the second tomographic image; and The second tomographic image is generated from the label image by using the learning model for generating the second tomographic image.

10. The image processing apparatus according to claim 1, further comprising: An image processing unit configured to identify a plurality of regions in at least one of the first tomographic image and the second tomographic image, wherein: The image processing unit recognizes a plurality of regions in the at least one image using the at least one image as input data to a learning model different from a learning model used to generate a second tomographic image.

11. The image processing apparatus according to claim 1, further comprising: an image processing unit configured to identify a plurality of regions in at least one of the first tomographic image and the second tomographic image; as well as A drive control unit is configured to drive and control an optical member that changes an imaging range so that any one of the identified plurality of areas is located at a predetermined position in the display area in a state where the second tomographic image is displayed as a moving image.

12. The image processing apparatus according to claim 1, further comprising: An image processing unit configured to identify a plurality of regions in at least one of the first tomographic image and the second tomographic image, wherein: The image processing unit generates an image analysis result for each of the identified multiple regions or a diagnosis result for each of the identified multiple regions by using the at least one image as input data to a learning model different from a learning model used to generate a second tomographic image.

13. The image processing apparatus according to claim 1, further comprising: An image processing unit configured to identify a plurality of regions in at least one of the first tomographic image and the second tomographic image, wherein: The image processing unit generates information about the difference between a tomographic image obtained using a generative adversarial network or an autoencoder for each of the identified multiple regions and the tomographic image input to the generative adversarial network or the autoencoder as information about the abnormal site.

14. The image processing apparatus according to claim 1, further comprising: An image processing unit configured to identify a plurality of regions in at least one of the first tomographic image and the second tomographic image, wherein: The image processing unit searches for a similar case image for each of the identified plurality of regions by using the at least one image as input data to a learning model different from a learning model used to generate the second tomographic image.

15. The image processing apparatus according to claim 1, wherein: The first tomographic image and the second tomographic image are three-dimensional OCT tomographic images; and The image processing apparatus further includes a generation unit configured to generate an en-face image corresponding to the local depth range of the second tomographic image.

16. The image processing apparatus according to claim 4, wherein: The training data includes: training data including a partial image as input data including a region of one of a vitreous body and a choroid in a tomographic image of an eye to be inspected obtained in an imaging mode corresponding to the one of the vitreous body and the choroid, and a partial image as correct answer data including a region of the one of the tomographic images obtained by subjecting the tomographic image obtained in the imaging mode to image processing for increasing contrast of the one region; training data including a partial image as input data including a region of the other of the vitreous body and the choroid in a tomographic image of the eye to be inspected obtained in an imaging mode corresponding to the other of the vitreous body and the choroid, and a partial image as correct answer data obtained by vertically inverting the partial image, the vertically inverted partial image including a region of the other of the vitreous body and the choroid in the tomographic image obtained by subjecting the tomographic image of the eye to be inspected obtained in the imaging mode corresponding to the other of the vitreous body and the choroid to image processing for improving the contrast of the region of the other; and Training data including a partial image as input data and a partial image as correct answer data, the partial image as input data including a retinal area in a tomographic image of an eye to be inspected obtained in an imaging mode corresponding to one of a vitreous body and a choroid, the partial image as correct answer data including a retinal area in a tomographic image obtained by subjecting the tomographic image obtained in the imaging mode to image processing for improving the contrast of the retinal area, the partial image as correct answer data being obtained by vertically inverting the partial image, the vertically inverted partial image including the retinal area in the tomographic image obtained by subjecting the tomographic image of the eye to be inspected obtained in an imaging mode corresponding to the other of a vitreous body and a choroid to image processing for improving the contrast of the retinal area.

17. The image processing apparatus according to claim 1, wherein: the third tomographic image and the fourth tomographic image have one of the plurality of regions but do not have another of the plurality of regions, and The fifth tomographic image and the sixth tomographic image have another region among the plurality of regions but do not have one region among the plurality of regions.

18. An image processing method, comprising: obtaining a first tomographic image of the eye to be inspected obtained by using optical coherence; as well as A second tomographic image having a plurality of regions including a retinal region and a vitreous region having a contrast higher than that of a plurality of regions in the first tomographic image obtained is generated using the obtained first tomographic image as input data input into a learning model, the learning model being obtained by using training data, the training data including a third tomographic image and a fourth tomographic image of the eye to be inspected obtained by using optical coherence, the third tomographic image having at least one region among a plurality of regions including a retinal region and a vitreous region, the fourth tomographic image having at least one region among the plurality of regions having a contrast higher than that of at least one region among the plurality of regions in the third tomographic image, and the training data including a fifth tomographic image and a sixth tomographic image of the eye to be inspected obtained by using optical coherence, the fifth tomographic image having at least another region among the plurality of regions, and the sixth tomographic image having at least another region among the plurality of regions having a contrast higher than that of at least another region among the plurality of regions in the fifth tomographic image.

19. A non-transitory computer-readable storage medium storing a program, which, when executed by a processor, causes the processor to execute the steps of the image processing method according to claim 18.

Images