Image processing device, image processing method, and computer readable medium

By using machine learning models to reduce noise and enhance contrast in OCT images, the problem of excessive noise or low contrast in OCT images after resolution is increased is solved, generating images that are more suitable for diagnosis and improving image quality and diagnostic accuracy.

CN112638234BActive Publication Date: 2024-11-05CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980057669.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-03-29
Filing Date
2019-06-14
Publication Date
2024-11-05
Estimated Expiration
2039-06-14

AI Technical Summary

Technical Problem

While existing OCT image processing technologies can improve resolution, they may also suffer from problems such as excessive noise or low contrast, making the images unsuitable for diagnosis.

Method used

Machine learning models are used to improve image quality, generating images more suitable for diagnosis through noise reduction and contrast enhancement. The original and improved images are switched, juxtaposed, or overlaid on the display unit by the display control unit to facilitate the identification of real tissue.

Benefits of technology

OCT images that are more suitable for image diagnosis are generated, noise is reduced and contrast is improved, helping operators to identify the authenticity of visualized tissues in the images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112638234B_ABST
    Figure CN112638234B_ABST
Patent Text Reader

Abstract

An image processing device includes: an image quality improvement unit, which is configured to generate a second image in which at least one of noise reduction and contrast enhancement is performed compared with the first image from a first image of a subject's eye by using a learning model; and a display control unit, which is configured to display the first image and the second image on a display unit by switching, arranging side by side, or superimposing the first image and the second image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a computer-readable medium. Background Art

[0002] An apparatus using optical coherence tomography (OCT) (OCT apparatus) has put into practical use a method for non-destructively and non-invasively obtaining a tomographic image of a subject such as a living body. In particular, the OCT apparatus is widely used as an ophthalmological apparatus for obtaining images for ophthalmic diagnosis.

[0003] In OCT, by interfering light reflected from the object to be measured with light reflected from a reference reflector and analyzing the intensity of the interfering light, a tomographic image of the object under test can be obtained. Time domain OCT (TD-OCT) is known as one of such OCTs. In TD-OCT, the depth information of the object under test is obtained by continuously changing the position of the reference reflector.

[0004] Spectral domain OCT (SD-OCT) and swept source OCT (SS-OCT) are also known. In SD-OCT, interference light obtained by causing light interference using low coherence light is divided, and frequency information is used instead of depth information to obtain frequency information. In SS-OCT, interference light is obtained by using light whose wavelength has been divided in advance using a wavelength scanning light source. Note that SD-OCT and SS-OCT are also collectively referred to as "Fourier domain OCT (FD-OCT)".

[0005] By using OCT, a tomographic image based on the depth information of the object under inspection can be obtained. In addition, by integrating the acquired three-dimensional tomographic image in the depth direction and projecting the integrated image onto a two-dimensional plane, a frontal image of the object under inspection can be generated. Conventionally, in order to improve the image quality of these images, images are acquired multiple times and averaged. However, in this case, it takes time to perform imaging multiple times.

[0006] Patent document 1 discloses a technology that converts a previously acquired image into an image with higher resolution by means of an artificial intelligence (AI) engine in order to cope with the rapid development in medical technology and also to meet the simple imaging in emergency situations. According to this technology, for example, an image acquired by performing imaging a small number of times can be converted into an image with higher resolution.

[0007] [Citation List]

[0008] [Patent Document]

[0009] Patent Document 1: Japanese Patent Application Publication No. 2018-5841 Summary of the invention

[0010] [Technical issues]

[0011] However, even if an image has high resolution, it cannot be said that the image is an image suitable for image diagnosis in some cases. For example, even when the image resolution is high, if there is a lot of noise or low contrast in the image, the object to be observed cannot be properly identified in some cases.

[0012] In this regard, an object of the present invention is to provide an image processing apparatus, an image processing method, and a computer-readable medium having a program stored thereon, the program being capable of generating an image more suitable for image diagnosis than conventional techniques.

[0013] [Solution to the problem]

[0014] According to one embodiment of the present invention, an image processing device includes: an image quality improvement unit, which is configured to use a learning model to generate a second image from a first image of a subject's eye that has undergone at least one of noise reduction and contrast enhancement compared to the first image; and a display control unit, which is configured to switch, juxtapose or superimpose the first image and the second image and display them on the display unit.

[0015] According to another embodiment of the present invention, an image processing device includes: an image quality improvement unit, which is constructed to generate a second image from a first image using a learning model, the first image being a frontal image generated based on information within a range of the subject's eye in the depth direction, the second image having undergone at least one of noise reduction and contrast enhancement compared to the first image; and a selection unit, which is constructed to select a learning model to be used by the image quality improvement unit from a plurality of learning models based on the range in the depth direction used to generate the first image.

[0016] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] [ Figure 1 ] is a diagram showing a schematic structure of an OCT device according to Example 1.

[0018] [ Figure 2 ] is a diagram showing a schematic structure of a control unit according to Example 1.

[0019] [ Figure 3A ] is a diagram showing an example of training data according to Example 1.

[0020] [ Figure 3B ] is a diagram showing an example of training data according to Example 1.

[0021] [ Figure 4] is a diagram showing an example of the construction of a learning model according to Example 1.

[0022] [ Figure 5 ] is a flowchart showing a series of image processing operations according to Example 1.

[0023] [ Fig. 6A ] is a diagram showing an example of a report screen in which display is switched between images obtained before and after image quality improvement processing.

[0024] [ Figure 6B ] is a diagram showing an example of a report screen in which display is switched between images obtained before and after image quality improvement processing.

[0025] [ Figure 7 ] is a diagram showing an example of a report screen in which images obtained before and after image quality improvement processing are displayed side by side.

[0026] [ Fig. 8A ] is a diagram showing an example of a report screen in which a plurality of images to which image quality improvement processing is applied are simultaneously displayed.

[0027] [ Figure 8B ] is a diagram showing an example of a report screen in which a plurality of images to which image quality improvement processing is applied are simultaneously displayed.

[0028] [ Fig. 9 ] is a diagram showing a schematic structure of a control unit according to Example 2.

[0029] [ Fig.10 ] is a flowchart showing a series of image processing operations according to Example 2.

[0030] [ Fig.11A ] is a diagram showing an example of changing the image quality improvement processing.

[0031] [ Fig. 11B ] is a diagram showing an example of changing the image quality improvement processing.

[0032] [ Fig. 12A ] is a diagram showing an example of a report screen in which a plurality of images to which image quality improvement processing is applied are simultaneously displayed.

[0033] [ Fig. 12B ] is a diagram showing an example of a report screen in which a plurality of images to which image quality improvement processing is applied are simultaneously displayed.

[0034] [ Fig.13 ] is a flowchart showing a series of image processing operations according to Example 3.

[0035] [ Fig.14] is a diagram showing a schematic structure of a control unit according to Example 4.

[0036] [ Fig.15 ] is a flowchart showing a series of image processing operations according to Example 4.

[0037] [ Fig.16A ] is a diagram showing an example of the construction of a neural network used as a machine learning model according to variation 9.

[0038] [ Fig. 16B ] is a diagram showing an example of the construction of a neural network used as a machine learning model according to variation 9.

[0039] [ Fig.17A ] is a diagram showing an example of the construction of a neural network used as a machine learning model according to variation 9.

[0040] [ Fig. 17B ] is a diagram showing an example of the construction of a neural network used as a machine learning model according to variation 9.

[0041] [ Fig.18 ] is a diagram showing an example of a user interface according to Example 5.

[0042] [ Fig.19A ] is a diagram showing an example of multiple OCTA en-face images.

[0043] [ Fig.19B ] is a diagram showing an example of multiple OCTA en-face images.

[0044] [ Fig. 20A ] is a diagram showing an example of a user interface according to Example 5.

[0045] [ Fig. 20B ] is a diagram showing an example of a user interface according to Example 5. DETAILED DESCRIPTION

[0046] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0047] However, the size, material, shape and relative position of the components described in the following embodiments are uncertain and may be changed according to the configuration of the device to which the present invention is applied or according to various conditions. In addition, in different drawings, the same or functionally similar elements are indicated by the same reference numerals.

[0048] In the following embodiments, although the eye to be inspected is given as an example of the object to be inspected, other organs of a person and the like may also be used as the object to be inspected. In addition, an OCTA (OCT angiography) image of the eye to be inspected is described as an example of an image that is processed for image quality improvement using a learning model related to a machine learning model (machine learning engine). Please note that the term "OCTA" refers to angiography using OCT without a contrast agent. In OCTA, an OCTA image (frontal vascular image) is generated by integrating three-dimensional motion contrast data obtained based on depth information of the object to be inspected in the depth direction and projecting the integrated data onto a two-dimensional plane.

[0049] Here, the term "motion contrast data" refers to data obtained by repeatedly imaging approximately the same position of an object under examination and detecting changes over time in the object during imaging. Note that the phrase "approximately the same position" refers to positions that are identical to the extent that motion contrast data can be generated, and includes positions that slightly deviate from the strictly identical position. Motion contrast data is obtained by calculating changes over time in the phase, vector, or intensity of a complex OCT signal, for example, based on a difference, ratio, or correlation.

[0050] Now, the main points related to the image quality improvement processing using the learning model related to the machine learning model will be mentioned. By performing image quality improvement processing on images using the learning model related to the machine learning model, although high-quality images can be obtained from a small number of images on the one hand, on the other hand, in some cases, tissues that do not actually exist are visualized in the image, or tissues that originally exist are not visualized in the image. Therefore, there is a problem that it is difficult to determine the authenticity of the visualized tissue in the image that has been improved by performing image quality improvement processing using the learning model.

[0051] Therefore, in the following embodiment, an image processing device is provided, which, by using a machine learning model, can generate an image that is more suitable for image diagnosis than conventional technology, and for such an image, the authenticity of the tissue visualized in the image can also be easily determined.

[0052] Note that although an OCTA image is described in the following embodiment, the image subjected to the image quality improvement process is not limited thereto and may be a tomographic image or an intensity en-face image, etc. Here, the term "en-face image" refers to a frontal image generated by projecting or integrating data within a predetermined depth range determined based on two reference planes onto a two-dimensional plane for three-dimensional data of the object under examination. Examples of en-face images include an intensity en-face image based on an intensity tomographic image and an OCTA image based on motion contrast data.

[0053] (Example 1)

[0054] In the following, reference is made to Figures 1 to 7 An optical coherence tomography apparatus (OCT apparatus) and an image processing method according to Embodiment 1 of the present invention will be described. Figure 1 is a schematic configuration of an OCT apparatus according to this embodiment.

[0055] The OCT apparatus 1 according to the present embodiment includes an OCT imaging unit 100 , a control unit (image processing apparatus) 200 , an input unit 260 , and a display unit 270 .

[0056] The OCT imaging unit 100 includes an imaging optical system of the SD-OCT apparatus, and acquires a signal including tomographic information of the subject's eye E based on interference light generated by causing return light from the subject's eye E irradiated with measurement light by a scanning unit and reference light corresponding to the measurement light to interfere with each other. An optical interference unit 110 and a scanning optical system 150 are provided in the OCT imaging unit 100.

[0057] The control unit 200 can control the OCT imaging unit 100, generate an image according to a signal obtained from the OCT imaging unit 100 or other devices (not shown), and process the generated / acquired image. The display unit 270 is any display such as an LCD display, and can display a GUI for operating the OCT imaging unit 100 and the control unit 200, a generated image, an image subjected to any type of processing, and various information such as patient information.

[0058] The input unit 260 is used to operate the control unit 200 by operating the GUI and by inputting information. The input unit 260 includes, for example, a pointing device such as a mouse, a touch pad, a trackball, a touch panel display, or a stylus pen, and a keyboard. Note that in the case of using a touch panel display, the display unit 270 and the input unit 260 may be configured integrally with each other. Note that although in the present embodiment, it is assumed that the OCT imaging unit 100, the control unit 200, the input unit 260, and the display unit 270 are units separated from each other, some or all of these units may be configured integrally with each other.

[0059] In the optical interference unit 110 in the OCT imaging unit 100, a light source 111, a coupler 113, a collimating optical system 121, a dispersion compensation optical system 122, a reflector 123, a lens 131, a diffraction grating 132, an imaging lens 133, and a line sensor 134 are provided. The light source 111 is a low-coherence light source that emits near-infrared light. The light emitted from the light source 111 propagates through the optical fiber 112a and enters the coupler 113 as a spectroscopic unit. The light entering the coupler 113 is divided into a measuring light and a reference light, and the measuring light travels to the scanning optical system 150 side, and the reference light travels to the reference light optical system side including the collimating optical system 121, the dispersion compensation optical system 122, and the reflector 123. The measuring light enters the optical fiber 112b and is guided to the scanning optical system 150. On the other hand, the reference light enters the optical fiber 112c and is introduced into the reference light optical system.

[0060] The reference light entering the optical fiber 112c is emitted from the optical fiber end, and is incident on the dispersion compensation optical system 122 through the collimating optical system 121, and is guided to the reflector 123. The reference light reflected by the reflector 123 travels along the optical path in the opposite direction and enters the optical fiber 112c again. The dispersion compensation optical system 122 is a component for compensating for the dispersion of the optical system relative to the scanning optical system 150 and the eye E to be inspected as the object to be inspected and matching the dispersion of the measurement light with the dispersion of the reference light. The reflector 123 is constructed to be drivable in the optical axis direction of the reference light by a driving unit (not shown) controlled by the control unit 200, and can relatively change the optical path length of the reference light relative to the optical path length of the measurement light and match the optical path lengths of the reference light and the measurement light.

[0061] On the other hand, the measuring light entering the optical fiber 112b is emitted from the optical fiber end and is incident on the scanning optical system 150. The scanning optical system 150 is an optical system configured to be relatively movable relative to the eye E to be inspected. The scanning optical system 150 is configured to be drivable in the front, back, upward, downward, left, and right directions relative to the eyeball axis of the eye E to be inspected by a driving unit (not shown) controlled by the control unit 200, and can be aligned relative to the eye E to be inspected. Note that the scanning optical system 150 can be configured to include a light source 111, a coupler 113, a reference light optical system, and the like.

[0062] The scanning optical system 150 is provided with a collimating optical system 151, a scanning unit 152, and a lens 153. Light emitted from the fiber end of the optical fiber 112b is substantially collimated by the collimating optical system 151, and is incident on the scanning unit 152.

[0063] The scanning unit 152 has two galvanometer mirrors capable of rotating the mirror surface, one of which deflects light in the horizontal direction, and the other deflects light in the vertical direction. The scanning unit 152 deflects the incident light according to the control of the control unit 200. In this way, the scanning unit 152 can scan the measurement light in two directions (i.e., the main scanning direction as a direction perpendicular to the paper surface (X direction) and the sub-scanning direction as a direction parallel to the paper surface (Y direction)) on the fundus Er of the eye E to be inspected. Note that the main scanning direction and the sub-scanning direction are not limited to the X direction and the Y direction, and it is sufficient as long as the main scanning direction and the sub-scanning direction are directions perpendicular to the depth direction (Z direction) of the eye E to be inspected and intersecting with each other. Therefore, for example, the main scanning direction may be the Y direction, and the sub-scanning direction may be the X direction.

[0064] The measurement light scanned by the scanning unit 152 forms an illumination spot on the fundus Er of the subject's eye E via the lens 153. When the scanning unit 152 receives in-plane deflection, each illumination spot moves (scans) on the fundus Er of the subject's eye E. The return light of the measurement light reflected and scattered from the fundus Er at the position of the illumination spot travels along an optical path in the opposite direction, enters the optical fiber 112 b, and returns to the coupler 113.

[0065] As described above, the reference light reflected by the reflector 123 and the return light of the measurement light from the fundus Er of the eye E to be inspected return to the coupler 113, and interfere with each other to become interference light. The interference light passes through the optical fiber 112d and is emitted to the lens 131. The interference light is substantially collimated by the lens 131 and is incident on the diffraction grating 132. The diffraction grating 132 has a periodic structure and separates the incident interference light. The separated interference light is imaged on the line sensor 134 through the imaging lens 133 that can change the focusing state. The line sensor 134 outputs a signal corresponding to the intensity of the light irradiated on each sensor unit to the control unit 200. The control unit 200 can generate a tomographic image of the eye to be inspected E based on the interference signal output from the line sensor 134.

[0066] Through the above-described series of operations, it is possible to obtain tomographic information regarding the depth direction at one point of the subject's eye E. Such a series of operations is called an "A scan."

[0067] In addition, by driving the galvanometer mirror of the scanning unit 152, interference light is generated at a point adjacent to the eye E to be inspected, and tomographic information in the depth direction at a point adjacent to the eye E to be inspected is acquired. By repeating this series of controls, multiple A scans are performed in an arbitrary lateral direction (main scanning direction), and two-dimensional tomographic information of the eye E to be inspected can be acquired in the aforementioned lateral direction and depth direction. This operation is called a "B scan". The control unit 200 can construct a B scan image by collecting multiple A scan images based on the interference signal acquired by the A scan. Hereinafter, the B scan image is referred to as a "two-dimensional tomographic image".

[0068] In addition, by slightly driving the galvanometer mirror of the scanning unit 152 in the sub-scanning direction orthogonal to the main scanning direction, it is possible to acquire tomographic information at another position (adjacent scanning line) of the eye to be inspected E. By repeating this operation to collect a plurality of B-scan images, the control unit 200 can acquire a three-dimensional tomographic image within a predetermined range of the eye to be inspected E.

[0069] Next, refer to Figure 2 The control unit 200 is described. Figure 2 2 shows a schematic configuration of the control unit 200. In the control unit 200, an obtaining unit 210, an image processing unit 220, a driving control unit 230, a memory 240, and a display control unit 250 are provided.

[0070] The acquisition unit 210 may acquire data of an output signal of the line sensor 134 corresponding to the interference signal of the subject's eye E from the OCT imaging unit 100. Note that the data of the output signal acquired by the acquisition unit 210 may be an analog signal or a digital signal. In the case where the acquisition unit 210 acquires an analog signal, the control unit 200 may convert the analog signal into a digital signal.

[0071] In addition, the acquisition unit 210 can obtain tomographic data generated by the image processing unit 220, and various images such as a two-dimensional tomographic image, a three-dimensional tomographic image, a motion contrast image, and an en-face image. Here, the term "tomographic data" refers to data including information related to a cross section of an object to be inspected, and includes a signal obtained by subjecting an interference signal obtained by OCT to Fourier transform, a signal obtained by subjecting a correlation signal to any processing, and a tomographic image based on these signals, and the like.

[0072] In addition, the acquisition unit 210 obtains a set of imaging conditions of the image to be subjected to image processing (for example, information related to imaging date and time, imaging part name, imaging area, imaging angle of view, imaging system, image resolution and grayscale, image size, image filter, and image data format). Note that the imaging condition group is not limited to the examples of the aforementioned imaging condition group. In addition, the imaging condition group does not necessarily include all the conditions mentioned in the aforementioned examples, and may include some of these conditions.

[0073] Specifically, the acquisition unit 210 acquires the imaging conditions of the OCT imaging unit 100 when the relevant image is imaged. In addition, the acquisition unit 210 may also acquire the imaging condition group stored in the data structure constituting the image according to the data format of the image. Please note that in the case where the imaging conditions are not stored in the data structure of the image, the acquisition unit 210 may also acquire the imaging information group including the imaging condition group from the storage device storing the imaging conditions, etc.

[0074] In addition, the obtaining unit 210 may also obtain information for identifying the subject's eye, such as a subject identification number, from the input unit 260, etc. Note that the obtaining unit 210 may obtain various data, various images, or various information from the memory 240 or other devices (not shown) connected to the control unit 200. The obtaining unit 210 may store the obtained various data or images in the memory 240.

[0075] The image processing unit 220 can generate a tomographic image or an en-face image, etc. from the data obtained by the obtaining unit 210 or the data stored in the memory 240, and can perform image processing on the generated or obtained image. The image processing unit 220 is provided with a tomographic image generating unit 221, a motion contrast generating unit 222, an en-face image generating unit 223, and an image quality improving unit 224.

[0076] The tomographic image generation unit 221 may subject the interference signal data obtained by the obtaining unit 210 to wave number conversion, Fourier transform, absolute value conversion (acquisition of amplitude), etc. to generate tomographic data, and may generate a tomographic image of the subject's eye E based on the tomographic data. The interference signal data obtained by the obtaining unit 210 may be data of a signal output from the line sensor 134, or may be data of an interference signal obtained from the memory 240 or a device (not shown) connected to the control unit 200. Note that any known method may be adopted as a method for generating a tomographic image, and a detailed description thereof is omitted here.

[0077] The tomographic image generation unit 221 can also generate a three-dimensional tomographic image based on the generated tomographic images of the plurality of parts. The tomographic image generation unit 221 can generate a three-dimensional tomographic image by, for example, arranging the tomographic images of the plurality of parts side by side in one coordinate system. Here, the tomographic image generation unit 221 can generate a three-dimensional tomographic image based on the tomographic images of the plurality of parts obtained from the memory 240 or a device (not shown) connected to the control unit 200.

[0078] The motion contrast generation unit 222 can generate a two-dimensional motion contrast image using a plurality of tomographic images obtained by imaging substantially the same position. In addition, the motion contrast generation unit 222 can generate a three-dimensional motion contrast image by arranging the generated two-dimensional motion contrast images of each part side by side in one coordinate system.

[0079] In the present embodiment, the motion contrast generation unit 222 generates a motion contrast image based on decorrelation values ​​between a plurality of tomographic images obtained by imaging substantially the same position of the eye E to be inspected.

[0080] Specifically, the motion contrast generation unit 222 acquires a plurality of tomographic images that have been aligned, with respect to a plurality of tomographic images that are obtained by continuously photographing approximately the same position at a photographing time. Note that various known methods can be used as the alignment method. For example, a reference image is selected from among a plurality of tomographic images, similarities with other tomographic images are calculated while changing the position and angle of the reference image, and the displacement amount of each tomographic image relative to the reference image is calculated. Then, alignment of the plurality of tomographic images is performed by correcting each tomographic image based on the calculation result. Note that the processing for alignment can be performed by a component separate from the motion contrast generation unit 222. In addition, the alignment method is not limited to this method, and alignment can be performed by any known method.

[0081] The motion contrast generation unit 222 calculates the decorrelation value of each two tomographic images whose imaging times are continuous with each other among the aligned plurality of tomographic images using the following Math. 1.

[0082] [Formula 1]

[0083]

[0084] Here, A(x, z) represents the amplitude at the position (x, z) of the tomographic image A, and B(x, z) represents the amplitude at the same position (x, z) of the tomographic image B. The decorrelation value M(x, z) obtained as a result takes values ​​from 0 to 1, and becomes closer to 1 as the difference between the two amplitude values ​​increases. Note that although the case of using a two-dimensional tomographic image on the XZ plane has been described in the present embodiment, a two-dimensional tomographic image on the YZ plane or the like may be used. In this case, the position (x, z) may be replaced by the position (y, z) or the like. Note that the decorrelation value may be determined based on the intensity value of the tomographic image, or may be determined based on the value of the interference signal corresponding to the tomographic image.

[0085] The motion contrast generation unit 222 determines the pixel values ​​of the motion contrast image based on the decorrelation values ​​M (x, z) at the respective positions (pixel positions), and generates the motion contrast image. Note that although in the present embodiment, the motion contrast generation unit 222 calculates the decorrelation values ​​for the tomographic images whose imaging times are continuous with each other, the method for calculating the motion contrast data is not limited thereto. The imaging times of the two tomographic images for which the decorrelation values ​​M are obtained do not have to be continuous with each other, and it is sufficient as long as the imaging times related to the tomographic images corresponding to each other are within a predetermined time interval. Therefore, for example, in order to extract an object that changes little over time, two tomographic images whose imaging interval is longer than the normal specified time can be extracted from the acquired multiple tomographic images, and the decorrelation values ​​can be calculated. In addition, instead of the decorrelation value, a variance value or a value obtained by dividing the maximum value by the minimum value (maximum value / minimum value), etc. can be determined.

[0086] Note that the method for generating a motion contrast image is not limited to the aforementioned method, and any other known method may also be used.

[0087] The en-face image generation unit 223 can generate an en-face image (OCTA image) as a frontal image from the three-dimensional motion contrast image generated by the motion contrast generation unit 222. Specifically, the en-face image generation unit 223 can generate an OCTA image as a frontal image by projecting the three-dimensional motion contrast image on a two-dimensional plane based on, for example, two arbitrary reference planes in the depth direction (Z direction) of the subject's eye E. In addition, the en-face image generation unit 223 can generate an intensity en-face image from the three-dimensional tomographic image generated by the tomographic image generation unit 221 in a similar manner.

[0088] More specifically, for example, the en-face image generation unit 223 determines a representative value of pixel values ​​in the depth direction at each position in the XY direction of the area surrounded by the two reference planes, determines the pixel values ​​at each position based on the representative value, and generates an en-face image. In this case, examples of the representative value include an average value, a median value, or a maximum value of pixel values ​​within a range in the depth direction of the area surrounded by the two reference planes.

[0089] Note that the reference plane may be a plane along the layer boundary at the cross section of the subject's eye E, or may be a plane. Hereinafter, the range in the depth direction between the reference planes used to generate the en-face image is referred to as an "en-face image generation range". In addition, the method for generating an en-face image according to the present embodiment is an example, and the en-face image generation unit 223 may generate an en-face image using any known method.

[0090] The image quality improvement unit 224 uses a learning model described later to generate a high-quality OCTA image based on the OCTA image generated by the en-face image generation unit 223. In addition, the image quality improvement unit 224 can generate a high-quality tomographic image or a high-quality intensity en-face image based on the tomographic image generated by the tomographic image generation unit 221 or the intensity en-face image generated by the en-face image generation unit 223. Note that the image quality improvement unit 224 can also generate a high-quality image based on various images obtained by the acquisition unit 210 from the memory 240 or other devices (not shown) connected to the control unit 200, not only based on the OCTA image captured using the OCT imaging unit 100 or the like. In addition, the image quality improvement unit 224 can perform image quality improvement processing on a three-dimensional motion contrast image or a three-dimensional tomographic image, not only on an OCTA image or a tomographic image.

[0091] The drive control unit 230 can control the driving of components such as the light source 111, the scanning optical system 150, the scanning unit 152, and the imaging lens 133 of the OCT imaging unit 100 connected to the control unit 200. The memory 240 can store various data obtained by the acquisition unit 210, and various data and images such as tomographic images or OCTA images generated and processed by the image processing unit 220. In addition, the memory 240 can store the attributes of the subject (name, age, etc.), information related to the subject's eye such as measurement results obtained using other inspection equipment (axial length of the eyeball, intraocular pressure, etc.), imaging parameters, image analysis parameters, and parameters set by the operator. Note that a configuration in which these images and information are stored in an external storage device (not shown) can also be adopted. The memory 240 can also store a program executed by the processor to realize the functions of each component of the control unit 200.

[0092] The display control unit 250 can cause various information obtained by the obtaining unit 210 and various images such as tomographic images, OCTA images, and three-dimensional motion contrast images generated and processed by the image processing unit 220 to be displayed on the display unit 270. The display control unit 250 can also cause information input by a user or the like to be displayed on the display unit 270.

[0093] The control unit 200 can be constituted, for example, by using a general-purpose computer. Note that the control unit 200 can be constituted by using a dedicated computer of the OCT device 1. The control unit 200 is equipped with a CPU (Central Processing Unit) (not shown) or an MPU (Micro Processing Unit) and a storage medium including a memory such as an optical disk or a ROM (Read Only Memory). Each component other than the memory 240 of the control unit 200 can be constituted by a software module executed by a processor such as a CPU or an MPU. In addition, the components discussed can be constituted by a circuit such as an ASIC that provides a specific function or an independent device. For example, the memory 240 can be constituted by any storage medium such as an optical disk or a memory.

[0094] Note that the control unit 200 may include one or more processors such as a CPU and a storage medium such as a ROM. Therefore, the various components of the control unit 200 may be configured to function when at least one or more processors and at least one storage medium are connected and at least one or more processors execute a program stored in at least one storage medium. Note that the processor is not limited to a CPU or an MPU, and may be a GPU (Graphics Processing Unit) or the like.

[0095] Next, refer to FIG. 3A to FIG. 4A learning model related to a machine learning model according to a machine learning algorithm such as deep learning according to the present embodiment is described. The learning model according to the present embodiment generates and outputs an image subjected to image quality improvement processing based on an input image according to a learning trend.

[0096] In this specification, the term "image quality improvement processing" refers to converting an input image into an image having an image quality more suitable for image diagnosis, and the term "high-quality image" refers to an image that has been converted into an image quality more suitable for image diagnosis. Here, the content of image quality suitable for image diagnosis depends on what is expected to be diagnosed using various image diagnoses. Therefore, although it cannot be generalized, for example, image quality suitable for image diagnosis includes the following image quality: low noise amount, high contrast, displaying the camera target in color and grayscale that makes the camera target easy to observe, large image size, and high resolution. In addition, image quality suitable for image diagnosis may include image quality such that objects or grayscales that do not actually exist that are rendered during the image generation process are removed from the image.

[0097] The term "learning model" refers to a model that has been trained (learned) in advance using appropriate training data for a machine learning model according to any machine learning algorithm such as deep learning. However, it is assumed that the learning model is not a model that does not perform further learning, and is a model that can also perform incremental learning. The training data consists of one or more pairs consisting of input data and a true value (correct answer data). In the present embodiment, a pair consisting of input data and a true value (ground truth) consists of an OCTA image and an OCTA image obtained by subjecting a plurality of OCTA images including the aforementioned OCTA image to an averaging process such as additive averaging.

[0098] The average image that has undergone the averaging process is a high-quality image suitable for image diagnosis because the pixels that are usually visualized in the source image group are enhanced. In this case, since the pixels that are usually visualized are enhanced, the generated high-quality image is a high-contrast image with a clear difference between low-intensity areas and high-intensity areas. In addition, for example, in the average image, random noise generated in each round of imaging can be reduced, and other source image groups can be used to make the areas that are not rendered well in the source image at a specific time point undergo interpolation processing.

[0099] Note that, among the pairs constituting the training data, pairs that do not contribute to improving the image quality can be removed from the training data. For example, if the image quality of a high-quality image as a true value included in a pair of training data is not suitable for image diagnosis, there is a possibility that an image output by a learning model learned using the relevant training data will have an image quality that is not suitable for image diagnosis. Therefore, by removing pairs whose image quality of the true value is not suitable for image diagnosis from the training data, the possibility that the learning model generates an image with an image quality that is not suitable for image diagnosis can be reduced.

[0100] Furthermore, in the case where the average intensity or intensity distribution differs greatly in a pair of images, there is a possibility that a learning model learned using the relevant training data will output an image that is not suitable for image diagnosis and has an intensity distribution that differs greatly from the intensity distribution of a low-quality image. Therefore, pairs of input data and true values ​​whose average intensity or intensity distribution differs greatly may be removed from the training data.

[0101] In addition, in the case where the structure or position of the imaging target to be rendered in the image group as a pair is greatly different, there is a possibility that the learning model learned using the relevant training data will output an image that is not suitable for image diagnosis and in which the structure or position of the imaging target to be rendered is greatly different from the low-quality image. Therefore, such a pair of input data and true value where the structure or position of the imaging target to be rendered is greatly different between the input data and the true value can also be removed from the training data.

[0102] By using the learning model learned in this manner, when an OCTA image obtained by one round of imaging (examination) is input, the image quality improvement unit 224 can generate a high-quality OCTA image with improved contrast or reduced noise, etc., by averaging processing. Therefore, the image quality improvement unit 224 can generate a high-quality image suitable for image diagnosis based on a low-quality image as an input image.

[0103] Next, the images used when learning are described. A pair of images is created using rectangular area images of a certain image size corresponding to each other in positional relationship, the pair consisting of an OCTA image 301 constituting training data and a high-quality OCTA image 302. Figure 3A and Figure 3B Describe the way in which the image in question was created.

[0104] First, a case is described in which a pair of training data is considered to be composed of an OCTA image 301 and a high-quality OCTA image 302. In this case, Figure 3AAs shown, a pair is formed in which the entire OCTA image 301 is considered as input data and the entire high-quality OCTA image 302 is considered as the ground truth. Note that although Figure 3A In the illustrated example, a pair consisting of input data and a true value is formed by using each image as a whole, but the pair is not limited to this.

[0105] For example, Figure 3B As shown, a pair may be formed in which a rectangular area image 311 in the OCTA image 301 is used as input data, and a rectangular area image 321 as a corresponding imaging area in the OCTA image 302 is used as a true value.

[0106] Note that when learning, the scan range (camera angle of view) and scan density (number of A scans and number of B scans) can be normalized to make the image size consistent, so that the rectangular area size can be consistent when learning. Figure 3A and Figure 3B The rectangular area image shown in is an example of the rectangular area sizes when the respective rectangular area sizes are used for learning individually from each other.

[0107] In addition, the number of rectangular areas in Figure 3A In the example shown in the Figure 3B In the example shown in FIG. 1 , a plurality of rectangular areas can be set. Figure 3B In the example shown, a pair can also be constructed in which a rectangular area image 312 in the OCTA image 301 is used as input data, and a rectangular area image 322 as a corresponding imaging area in the high-quality OCTA image 302 is used as a true value. Therefore, a rectangular area image pair different from each other can be created from a pair consisting of one OCTA image and one high-quality OCTA image. Note that the content of the pair group constituting the training data can be enhanced by creating a large number of rectangular area image pairs while changing the position of the area to different coordinates in the OCTA image as the source image and the high-quality OCTA image.

[0108] Despite Figure 3BIn the example shown in , the rectangular areas are discretely shown, but the OCTA image as the source image and the high-quality OCTA image can each be continuously and seamlessly divided into a group of rectangular area images with a consistent image size. Alternatively, the OCTA image as the source image and the high-quality OCTA image can each be divided into rectangular area image groups at random positions corresponding to each other. In this way, by selecting an image of a smaller area as a pair consisting of input data as a rectangular area and a true value, a large amount of pair data can be generated from the OCTA image 301 and the high-quality OCTA image 302 constituting the original pair. Therefore, the time required to train the machine learning model can be shortened.

[0109] Next, refer to Figure 4 A convolutional neural network (CNN) that performs image quality improvement processing with respect to an input tomographic image is described as one example of a learning model according to the present embodiment. Figure 4 An example of a construction 401 of a learning model used by the image quality improving unit 224 is shown.

[0110] exist Figure 4 The learning model shown in 401 is composed of a plurality of layer groups responsible for processing a group of input values ​​for output. Note that the types of layers included in the construction 401 of the learning model are convolutional layers, downsampling layers, upsampling layers, and pooling layers.

[0111] The convolution layer is a layer that performs convolution processing on the input value group according to the set parameters such as the kernel size of the filter, the number of filters, the stride value, and the dilation value. Please note that the dimension of the kernel size of the filter can be changed according to the dimension of the input image.

[0112] The down-sampling layer is a layer that performs processing such that the number of output value groups is smaller than the number of input value groups by thinning or combining input value groups. Specifically, for example, a maximum pooling process can be used as such a process.

[0113] The up-sampling layer is a layer that performs processing such that the number of output value groups is larger than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. Specifically, for example, a linear interpolation process can be used as such a process.

[0114] The merging layer is a layer that processes by inputting value groups such as an output value group of a certain layer or a pixel value group constituting an image from multiple sources and merging the value groups by concatenating or adding the value groups.

[0115] Please note that as a Figure 4The parameters set for the convolution layer group included in the illustrated configuration 401, for example, by setting the kernel size of the filter to a width of 3 pixels and a height of 3 pixels and setting the number of filters to 64, can perform image quality improvement processing with a certain degree of accuracy. However, it is important to be careful in this regard because if the settings of the parameters regarding the layer group and the node group constituting the neural network are different, then in some cases, the degree to which the trend trained based on the training data can be reproduced in the output data will be different. In other words, in many cases, the appropriate parameters will differ depending on the form at the time of implementation, so the parameters can be changed to preferred values ​​as needed.

[0116] In addition, there are cases where CNN can obtain better characteristics by changing the construction of CNN, rather than simply by using the method of changing parameters as described above. The term "better characteristics" refers to, for example, increased accuracy of image quality improvement processing, reduced time spent on image quality improvement processing, and reduced time required for training a machine learning model.

[0117] Although not shown in the drawings, as a modification example of the configuration of CNN, for example, a batch normalization layer or an activation layer using a rectified linear unit may be incorporated after a convolution layer or the like.

[0118] When data is input into the learning model of such a machine learning model, data according to the design of the machine learning model is output. For example, according to the tendency of training the machine learning model using the training data, output data having a high probability corresponding to the input data is output. In the case of the learning model according to the present embodiment, when an OCTA image 301 is input, according to the tendency of training the machine learning model using the training data, a high-quality OCTA image 302 is output.

[0119] Note that in the case where learning is performed in a manner in which the image is divided into regions, the learning model outputs rectangular region images that are high-quality OCTA images corresponding to the respective rectangular regions. In this case, first, the image quality improvement unit 224 divides the OCTA image 301 that is the input image into rectangular region image groups based on the image size when learning is performed, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improvement unit 224 arranges the respective images of a group of rectangular region images that are high-quality OCTA images output from the learning model according to the same positional relationship as the positional relationship of the respective images of the rectangular region image group that is input into the learning model, and merges the rectangular region images. In this way, the image quality improvement unit 224 can generate a high-quality OCTA image 302 that corresponds to the input OCTA image 301.

[0120] Next, refer to Figures 5 to 7 A series of image processing operations according to the present embodiment will be described. Figure 5 is a flowchart showing a series of image processing operations according to the present embodiment.

[0121] First, in step S501, the acquisition unit 210 acquires a plurality of three-dimensional tomographic information obtained by imaging the subject's eye E a plurality of times. The acquisition unit 210 may acquire the tomographic information of the subject's eye E using the OCT imaging unit 100, or may acquire the tomographic information from the memory 240 or other devices connected to the control unit 200.

[0122] Here, a case where tomographic information of the subject's eye E is obtained by using the OCT imaging unit 100 will be described. First, the operator makes a patient as a subject sit in front of the scanning optical system 150, performs alignment, inputs patient information into the control unit 200, etc., and then starts OCT imaging. The drive control unit 230 of the control unit 200 drives the galvanometer mirror of the scanning unit 152 to scan approximately the same position of the subject's eye multiple times, thereby obtaining multiple items of tomographic information (interference signals) at approximately the same position of the subject's eye. Thereafter, the drive control unit 230 slightly drives the galvanometer mirror of the scanning unit 152 in a sub-scanning direction orthogonal to the main scanning direction, and obtains multiple items of tomographic information at other positions (adjacent scanning lines) of the subject's eye E. By repeating this control, the obtaining unit 210 obtains multiple items of three-dimensional tomographic information within a predetermined range of the subject's eye E.

[0123] Next, in step S502, the tomographic image generation unit 221 generates a plurality of three-dimensional tomographic images based on the obtained plurality of three-dimensional tomographic information items. Note that in step S501, when the acquisition unit 210 acquires a plurality of three-dimensional tomographic images from the memory 240 or other devices connected to the control unit 200, step S502 may be omitted.

[0124] In step S503, the motion contrast generation unit 222 generates three-dimensional motion contrast data (three-dimensional motion contrast image) based on the plurality of three-dimensional tomographic images. Note that the motion contrast generation unit 222 may obtain a plurality of motion contrast data based on three or more tomographic images acquired about substantially the same position, and generate an average value of the plurality of motion contrast data as final motion contrast data. Note that in the case where the acquisition unit 210 obtains three-dimensional motion contrast data from the memory 240 or other device connected to the control unit 200 in step S501, steps S502 and S503 may be omitted.

[0125] In step S504, the en-face image generation unit 223 generates an OCTA image for the three-dimensional motion contrast data according to an instruction from the operator or based on a predetermined en-face image generation range. Note that in the case where the acquisition unit 210 acquires an OCTA image from the memory 240 or other device connected to the control unit 200 in step S501, steps S502 to S504 may be omitted.

[0126] In step S505, the image quality improvement unit 224 performs image quality improvement processing on the OCTA image using the learning model. The image quality improvement unit 224 inputs the OCTA image into the learning model, and generates a high-quality OCTA image based on the output from the learning model. Note that in the case where the learning model is learned in a manner that divides the image into regions, the image quality improvement unit 224 first divides the OCTA image as the input image into rectangular region image groups based on the image size at the time of learning, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improvement unit 224 arranges each image of a group of rectangular region images as the high-quality OCTA image output from the learning model according to the same positional relationship as the positional relationship of each image of the rectangular region image group input into the learning model, and merges these rectangular region images, thereby generating a final high-quality OCTA image.

[0127] In step S506, the display control unit 250 causes the display unit 270 to switch from displaying the original OCTA image (first image) to displaying the high-quality OCTA image (second image) generated by the image quality improvement unit 224. As described above, in the image quality improvement process using the machine learning model, in some cases, a blood vessel that does not actually exist is visualized in the OCTA image, or a blood vessel that originally existed is not visualized in the OCTA image. In this regard, by causing the display control unit 250 to switch the display unit 270 from displaying the original OCTA image to displaying the generated high-quality OCTA image, it is possible to facilitate determination of whether the blood vessel is a blood vessel newly generated by the image quality improvement process or a blood vessel that also exists in the original image. When the display process of the display control unit 250 ends, a series of image processing operations ends.

[0128] Next, refer to FIG. 6A to FIG. 7 A method for operating the control unit 200 is described. Fig. 6A and Figure 6B An example of a report screen that switches display between images before and after image quality improvement processing is shown. Fig. 6A The report screen 600 shown in FIG. 6 shows a tomographic image 611 and an OCTA image 601 before image quality improvement processing. Figure 6BA tomographic image 611 and an OCTA image 602 (high-quality OCTA image) after image quality improvement processing are shown on the report screen 600 shown.

[0129] exist Fig. 6A On the report screen 600 shown, when the operator uses a mouse as an example of the input unit 260 and presses the right button of the mouse on the OCTA image 601, a popup menu 620 for selecting whether to perform the image quality improvement process is displayed. When the operator selects to perform the image quality improvement process on the popup menu 620, the image quality improvement unit 224 performs the image quality improvement process on the OCTA image 601.

[0130] Then, if Figure 6B As shown, the display control unit 250 switches the display on the report screen 600 from displaying the OCTA image 601 before the image quality improvement process to displaying the image OCTA 602 after the image quality improvement process. Note that it is also possible to open the pop-up menu 620 by pressing the right button of the mouse on the OCTA image 602 again, and switch the display to displaying the OCTA image 601 before the image quality improvement process.

[0131] Note that although an example has been described in which the display is switched between the images before and after the image quality improvement process by using the pop-up menu 620 displayed in accordance with the operator's operation of pressing the right button of the mouse, any other method other than the pop-up menu may be performed as a method of switching images. For example, a button provided on the report screen (e.g., Fig.18 , Fig. 20A and Fig. 20B The image switching can be performed by the button 3420 shown in FIG. 34 ), a pull-down menu, a radio button, a check box or a keyboard operation. In addition, the display image can be switched by the operation of a mouse wheel or a touch operation on a touch panel display.

[0132] The operator can arbitrarily switch between displaying the OCTA image 601 before the image quality improvement process and the OCTA image 602 after the image quality improvement process by the above method. Therefore, the operator can easily view and compare the OCTA images before and after the image quality improvement process, and can easily confirm the changes between the OCTA images caused by the image quality improvement process. Therefore, the operator can easily identify the blood vessels that are actually not present and are visualized in the OCTA image by the image quality improvement process, or the blood vessels that originally existed and disappeared from the OCTA image due to the image quality improvement process, and can easily confirm the authenticity of the tissue visualized in the image.

[0133] Note that although in the above-described display method, the images before and after the image quality improvement processing are switched and displayed, a similar effect can be obtained by displaying these images in a juxtaposed or superimposed manner. Figure 7 An example of a report screen in the case where images before and after image quality improvement processing are displayed in a juxtaposed manner is shown. Figure 7 On the report screen 700 shown, an OCTA image 701 before the image quality improvement process and an OCTA image 702 after the image quality improvement process are displayed in a juxtaposed manner.

[0134] Also in this case, the operator can easily view and compare the images before and after the image quality improvement process, and can easily confirm the changes between the images caused by the image quality improvement process. Therefore, the operator can easily identify the blood vessels that are actually not present and are visualized in the OCTA image by the image quality improvement process, or the originally existing blood vessels that disappear from the OCTA image due to the image quality improvement process, and can easily determine the authenticity of the tissue visualized in the image. Note that in the case of displaying the images before and after the image quality improvement process in a superimposed manner, the display control unit 250 can set transparency for at least one of the images before and after the image quality improvement process, and display the images before and after the image quality improvement process in a superimposed manner on the display unit 270.

[0135] In addition, as described above, the image quality improvement unit 224 can also perform image quality improvement processing using a learning model on a tomographic image or an intensity en-face image other than an OCTA image. In this case, as a pair of training data for the learning model, a pair can be used in which a tomographic image or an intensity en-face image before averaging is used as input data, and a tomographic image or an intensity en-face image after averaging is used as a true value. Note that in this case, the learning model may be a single learning model that has been learned using training data such as an OCTA image or a tomographic image, or a plurality of learning models that have been learned for respective types of images may be used as the learning model. In the case of using a plurality of learning models, the image quality improvement unit 224 may use a learning model corresponding to the type of image that is the object of the image quality improvement processing. Note that the image quality improvement unit 224 may perform image quality improvement processing using a learning model for a three-dimensional motion contrast image or a three-dimensional tomographic image, and the training data in this case may also be prepared in the same manner as described above.

[0136] exist Figure 7, a tomographic image 711 before image quality improvement processing and a tomographic image 712 after image quality improvement processing are displayed in a juxtaposed manner. Fig. 6A and Figure 6B , the display control unit 250 can cause the tomographic images or intensity en-face images before and after the image quality improvement processing to be switched and displayed on the display unit 270. In addition, the display control unit 250 can cause the tomographic images or intensity en-face images before and after the image quality improvement processing to be displayed on the display unit 270 in a superimposed manner. Also in these cases, the operator can easily view and compare the images before and after the image quality improvement processing, and can easily confirm the changes between the images caused by the image quality improvement processing. Therefore, the operator can easily identify the tissue that does not actually exist but is visualized in the image by the image quality improvement processing, or the tissue that originally existed but disappeared from the image due to the image quality improvement processing, and can easily determine the authenticity of the tissue visualized in the image.

[0137] As described above, the control unit 200 according to the present embodiment includes the image quality improvement unit 224 and the display control unit 250. The image quality improvement unit 224 generates a second image that has undergone at least one of noise reduction and contrast enhancement compared to the first image from the first image of the subject's eye using a learning model. The display control unit 250 causes the first image and the second image to be switched, juxtaposed, or superimposed and displayed on the display unit 270. Note that the display control unit 250 can switch between the first image and the second image and display the related switched image on the display unit 270 according to an instruction from the operator.

[0138] In this way, the control unit 200 can generate a high-quality image with reduced noise and / or enhanced contrast from the source image. Therefore, compared with conventional techniques, the control unit 200 can generate an image more suitable for image diagnosis, such as a clearer image or an image in which the part or lesion desired to be observed is enhanced.

[0139] Furthermore, the operator can easily view and compare images before and after the image quality improvement process, and can easily confirm changes between images caused by the image quality improvement process. Therefore, the operator can easily identify tissue that is actually not present but is visualized in the image by the image quality improvement process, or tissue that originally existed but has disappeared from the image due to the image quality improvement process, and can easily confirm the authenticity of the tissue visualized in the image.

[0140] Although the average image is used as the true value of the training data for learning the model according to the present embodiment, the training data is not limited thereto. For example, a high-quality image obtained by performing maximum a posteriori processing (MAP estimation processing) on ​​the source image group can be used as the true value of the training data. In the MAP estimation processing, a likelihood function is obtained based on the probability density of each pixel value in a plurality of images, and the obtained likelihood function is used to estimate the true signal value (pixel value).

[0141] The high-quality image obtained by the MAP estimation process is a high-contrast image based on a pixel value close to the true signal value. In addition, since the estimated signal value is determined based on the probability density, the randomly generated noise is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as training data, the learning model can generate a high-quality image with reduced noise and high contrast suitable for image diagnosis from the input image. Note that regarding the method of generating pairs of input data and true values ​​for training data, a method similar to the case of using an average image as training data can be performed.

[0142] In addition, a high-quality image obtained by applying a smoothing filter process to a source image can be used as a true value of the training data. In this case, the learning model can generate a high-quality image with reduced random noise from the input image. In addition, an image obtained by applying a grayscale conversion process to a source image can also be used as a true value of the training data. In this case, the learning model can generate a high-quality image with enhanced contrast from the input image. Note that regarding the method for generating a pair of input data and a true value for training data, a method similar to the case of using an average image as training data can be performed.

[0143] Note that the input data of the training data may be an image obtained from an imaging device having the same image quality tendency as that of the OCT imaging unit 100. In addition, the true value of the training data may be a high-quality image obtained by a high-cost process such as a process using a successive approximation method, or may be a high-quality image obtained by imaging the object corresponding to the input data using an imaging device having a higher performance than the OCT imaging unit 100. In addition, the true value may be a high-quality image obtained by performing a rule-based noise reduction process based on the structure of the object to be inspected, etc. Here, the noise reduction process may include, for example, a process of replacing a high-intensity pixel of only one pixel, which is obviously noise appearing in a low-intensity area, with an average value of adjacent low-intensity pixel values. Therefore, the learning model may adopt, as training data, an image captured by an imaging device having a higher performance than that used to capture the input image, or an image obtained by an imaging process involving a larger number of processes than the imaging process used to obtain the input image.

[0144] Note that although the image quality improvement unit 224 has been described as generating a high-quality image with reduced noise or enhanced contrast by using a learning model, the image quality improvement processing performed by the image quality improvement unit 224 is not limited to this. As described above, it is sufficient that the image quality improvement unit 224 can generate an image with image quality more suitable for image diagnosis through the image quality improvement processing.

[0145] Furthermore, in the case where the images before and after the image quality improvement process are displayed in a juxtaposed manner on the display unit 270, the display control unit 250 may, according to an instruction from the operator, enlarge and display any image among the images before and after the image quality improvement process being displayed in a juxtaposed manner on the display unit 270. More specifically, for example, Figure 7 , if the operator selects an OCTA image 701, the display control unit 250 can display the OCTA image 701 in an enlarged manner on the report screen 700. Furthermore, if the operator selects an OCTA image 702 after the image quality improvement process, the display control unit 250 can display the OCTA image 702 in an enlarged manner on the report screen 700. In this case, the operator can observe in more detail the image that the operator wants to observe among the images before and after the image quality improvement process.

[0146] In addition, in the case where the generation range of an en-face image such as an OCTA image is changed according to the operator's instruction, the control unit 200 can change the display from the image displayed in a superimposed manner to the image based on the changed generation range and the image that has undergone image quality improvement. More specifically, when the operator changes the en-face image generation range through the input unit 260, the en-face image generation unit 223 generates an en-face image before the image quality improvement process based on the changed generation range. The image quality improvement unit 224 generates a high-quality en-face image from the en-face image newly generated by the en-face image generation unit 223 using the learning model. Thereafter, the display control unit 250 causes the display unit 270 to change from displaying the en-face images before and after the image quality improvement process that are being displayed in a juxtaposed manner to displaying the newly generated en-face images before and after the image quality improvement process. In this case, while the operator arbitrarily changes the range in the depth direction that the operator wishes to observe, the operator can observe the en-face images before and after the image quality improvement process based on the changed range in the depth direction.

[0147] (Variant 1)

[0148] As described above, in an image that has been processed for image quality improvement using a learning model, tissue that does not actually exist may be visualized, or tissue that originally existed may not be visualized. Therefore, misdiagnosis may occur due to the operator performing image diagnosis based on such an image. Therefore, when an OCTA image or a tomographic image, etc. after image quality improvement processing is displayed on the display unit 270, the display control unit 250 may also display information to the effect that the image in question is an image that has been processed for image quality improvement using a learning model. In this case, misdiagnosis by the operator can be suppressed. Note that the display form may be in any form as long as the form makes it understand that the image is a high-quality image obtained using a learning model.

[0149] (Variant 2)

[0150] In Embodiment 1, an example in which image quality improvement processing is applied to an OCTA image or tomographic image, etc., obtained by one round of imaging (examination) is described. In this regard, image quality improvement processing using a learning model can also be applied to a plurality of OCTA images or tomographic images, etc., obtained by performing imaging (examination) a plurality of times. Fig. 8A and Figure 8B A configuration is described in which images obtained by applying image quality improvement processing using a learning model to a plurality of OCTA images, tomographic images, or the like are simultaneously displayed.

[0151] Fig. 8A and Figure 8B Each shows an example of a time series report screen for displaying a plurality of OCTA images obtained by imaging the same eye to be inspected a plurality of times over time. Fig. 8A On the report screen 800 shown in FIG. 8 , a plurality of OCTA images 801 before image quality improvement processing are displayed side by side in chronological order. The report screen 800 also includes a pop-up menu 820 , and the operator can select whether to apply image quality improvement processing by operating the pop-up menu 820 via the input unit 260 .

[0152] If the operator chooses to apply the image quality improvement process, the image quality improvement unit 224 applies the image quality improvement process using the learning model to all OCTA images being displayed. Figure 8B As shown, the display control unit 250 switches from displaying the plurality of OCTA images 801 being displayed to displaying the plurality of OCTA images 802 after image quality improvement processing has been performed.

[0153] Furthermore, if the operator selects not to apply the image quality improvement process on the popup menu 820 , the display control unit 250 switches from displaying the plurality of OCTA images 802 after the image quality improvement process being displayed to displaying the plurality of OCTA images 801 before the image quality improvement process.

[0154] Note that in this modification, an example of simultaneously switching and displaying a plurality of OCTA images obtained before and after the image quality improvement process using the learning model has been described. However, it is possible to simultaneously switch and display a plurality of tomographic images or intensity en-face images obtained before and after the image quality improvement process using the learning model. Note that the operation method is not limited to the method using the pop-up menu 820, and any operation method may be adopted, for example, a method using a button, a pull-down menu, a radio button, or a check box provided on the report screen, or an operation for a keyboard, a mouse wheel, or a touch panel.

[0155] (Example 2)

[0156] The learning model outputs output data having a high probability of corresponding to the input data according to the learning trend. In this regard, when the learning model uses a group of images having similar image quality trends to each other as training data for learning, images that have undergone more effective image quality improvement can be output for images having the similar trends in question. Therefore, in Embodiment 2, the image quality improvement process is performed more effectively by performing the image quality improvement process using a plurality of learning models, wherein the plurality of learning models undergo learning using training data consisting of pairs of groups that are grouped for each imaging condition such as an imaging site or for each en-face image generation range.

[0157] In the following, reference is made to Fig. 9 and Fig.10 The OCT apparatus according to the present embodiment is described below. Note that since the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to Embodiment 1 except for the control unit, Figure 1 The same components as those shown in the embodiment are denoted by the same reference numerals as those in the embodiment 1, and descriptions of these components are omitted hereinafter. Hereinafter, the OCT apparatus according to the present embodiment will be described centering on the differences from the OCT apparatus 1 according to the embodiment 1.

[0158] Fig. 9 2 shows a schematic configuration of a control unit 900 according to the present embodiment. Note that the components of the control unit 900 according to the present embodiment are the same as those of the control unit 200 according to Embodiment 1 except for the image processing unit 920 and the selection unit 925. Figure 2 The same components as those shown in are denoted by the same reference numerals as those in Embodiment 1, and description of these components is omitted hereinafter.

[0159] In addition to the tomographic image generating unit 221 , the motion contrast generating unit 222 , the en-face image generating unit 223 , and the image quality improving unit 224 , a selecting unit 925 is provided in the image processing unit 920 of the control unit 900 .

[0160] The selection unit 925 selects a learning model to be used by the image quality improvement unit 224 from among a plurality of learning models based on the imaging conditions or the en-face image generation range of the image to be subjected to the image quality improvement processing by the image quality improvement unit 224. The image quality improvement unit 224 performs the image quality improvement processing on the target OCTA image or tomographic image, etc., using the learning model selected by the selection unit 925, and generates a high-quality OCTA image or a high-quality tomographic image.

[0161] Next, a plurality of learning models according to the present embodiment will be described. As described above, the learning model outputs output data having a high probability of corresponding to the input data according to the learning trend. In this regard, when the learning model uses a group of images having similar image quality trends to each other as training data for learning, images that have undergone more effective image quality improvement can be output for images having the similar trends in question. Therefore, in the present embodiment, a plurality of learning models are prepared, and the plurality of learning models undergo learning using training data consisting of pairs of groups that are grouped according to imaging conditions including conditions such as imaging sites, imaging systems, imaging areas, imaging angles, scanning density, and image resolution, or for respective en-face image generation ranges.

[0162] More specifically, for example, a plurality of learning models are prepared, such as a learning model that uses an OCTA image with the macula region set as an imaging site as training data, and a learning model that uses an OCTA image with the optic nerve head set as an imaging site as training data. Note that the macula region and the optic nerve head are each an example of an imaging site, and other imaging sites may be included. In addition, a learning model may be prepared for which an OCTA image for each specific imaging site such as the macula region or the optic nerve head is used as training data.

[0163] In addition, for example, the visualization of structures such as blood vessels visualized in OCTA images differs significantly between the case where the retina is imaged at a wide angle of view and low density and the case where the retina is imaged at a narrow angle of view and high density. Therefore, a learning model that learns each set of training data according to the imaging angle of view and the scanning density can be prepared. In addition, examples of imaging systems include SD-OCT imaging systems and SS-OCT imaging systems, and image quality, imaging range, penetration depth in the depth direction, and the like differ according to the differences between these imaging systems. Therefore, a learning model that learns using training data according to the respective types of imaging systems can be prepared.

[0164] In addition, it is usually rare to generate an OCTA image that extracts blood vessels of all layers of the retina at once, and it is usually generated to extract only blood vessels existing in a predetermined depth range. For example, for depth ranges such as the superficial, deep, and outer layers of the retina and the superficial choroid layer, an OCTA image is generated in which blood vessels are extracted in each depth range. On the other hand, the form of blood vessels visualized in the OCTA image greatly differs depending on the depth range. For example, blood vessels visualized in the superficial layer of the retina form a low-density, thin, and clear vascular network, while blood vessels visualized in the superficial choroid layer are visualized with high density, and it is difficult to clearly distinguish individual blood vessels. Therefore, a learning model that is learned using respective training data sets according to the generation range of en-face images such as OCTA images can be prepared.

[0165] Although an example using OCTA images as training data is described here, similar to Embodiment 1, in the case of performing image quality improvement processing on tomographic images or intensity en-face images, these images can be used as training data. In this case, a plurality of learning models are prepared for learning using respective training data sets according to the imaging conditions of these images or the en-face image generation range.

[0166] Next, refer to Fig.10 A series of image processing operations according to the present embodiment is described. Fig.10 1 is a flowchart showing a series of image processing operations according to the present embodiment. Note that descriptions about the same processes as those in the series of image processing operations according to Embodiment 1 are omitted as appropriate.

[0167] First, in step S1001, similarly to step S501 according to Embodiment 1, the obtaining unit 210 obtains a plurality of pieces of three-dimensional tomographic information obtained by performing a plurality of imaging operations on the subject's eye E. The obtaining unit 210 may obtain the tomographic information of the subject's eye E using the OCT imaging unit 100, or may obtain the tomographic information from the memory 240 or other devices connected to the control unit 200.

[0168] The acquisition unit 210 also acquires a group of imaging conditions related to the tomographic information. Specifically, when performing imaging related to the tomographic information, the acquisition unit 210 can acquire imaging conditions such as an imaging part and an imaging system. Note that, depending on the data format of the tomographic information, the acquisition unit 210 can acquire a group of imaging conditions stored in a data structure of data constituting the tomographic information. In addition, in the case where the imaging conditions are not stored in the data structure of the tomographic information, the acquisition unit 210 can acquire the imaging information group from a server or a database or the like storing a file describing the imaging conditions. In addition, the acquisition unit 210 can estimate the imaging information group from an image based on the tomographic information by any known method.

[0169] Furthermore, in the case where the acquisition unit 210 acquires a plurality of three-dimensional tomographic images, a plurality of three-dimensional motion contrast data, or a plurality of OCTA images, etc., the acquisition unit 210 acquires an imaging condition group related to the acquired images or data. Note that in the case where the image quality improvement process is performed only using a plurality of learning models that are learned using the respective kinds of training data according to the generation range of the OCTA image or the intensity en-face image, the acquisition unit 210 does not need to acquire an imaging condition group for the tomographic images.

[0170] The processing of step S1002 to step S1004 is the same as the processing of step S502 to S504 according to Embodiment 1, and thus the description of the processing is omitted here. When the en-face image generation unit 223 generates an OCTA image in step S1004, the process proceeds to step S1005.

[0171] In step S1005, based on the imaging condition group or the generation range related to the generated OCTA image and the information on the training data related to the plurality of learning models, the selection unit 925 selects the learning model used by the image quality improvement unit 224. More specifically, for example, in the case where the imaging part in the OCTA image is the optic nerve head, the selection unit 925 selects the learning model that has been learned using the OCTA image of the optic nerve head as the training data. Furthermore, for example, in the case where the generation range of the OCTA image is the superficial layer of the retina, the selection unit 925 selects the learning model that has been learned using the OCTA image with the superficial layer of the retina set as the generation range as the training data.

[0172] Note that even if the imaging condition group or generation range related to the generated OCTA image and the information related to the training data of the learning model do not completely match, the selection unit 925 may select a learning model that has been learned using images having a similar trend in image quality as training data. In this case, for example, the selection unit 925 may include a table that describes the correlation between the imaging condition group or generation range related to the OCTA image and the learning model to be used.

[0173] In step S1006, the image quality improvement unit 224 performs image quality improvement processing on the OCTA image generated in step S1004 using the learning model selected by the selection unit 925, thereby generating a high-quality OCTA image. The method for generating a high-quality OCTA image is the same as in step S505 according to Embodiment 1, so its description is omitted here.

[0174] Step S1007 is the same as step S506 according to Embodiment 1, and thus description thereof is omitted here. When the high-quality OCTA image is displayed on the display unit 270 in step S1007, a series of image processing operations according to the present embodiment ends.

[0175] As described above, the control unit 900 according to the present embodiment includes the selection unit 925 that selects, from a plurality of learning models, a learning model to be used by the image quality improvement unit 224. The selection unit 925 selects the learning model to be used by the image quality improvement unit 224 based on a range in the depth direction for generating an OCTA image to be subjected to image quality improvement processing.

[0176] For example, the selection unit 925 may select a learning model based on a display portion in an OCTA image to be subjected to image quality improvement processing and a range in a depth direction for generating an OCTA image. Also, for example, the selection unit 925 may select a learning model to be used by the image quality improvement unit 224 based on an imaging portion including a display portion in an OCTA image to be subjected to image quality improvement processing and a range in a depth direction for generating an OCTA image. Also, for example, the selection unit 925 may select a learning model to be used by the image quality improvement unit 224 based on imaging conditions of an OCTA image to be subjected to image quality improvement processing.

[0177] Therefore, by performing image quality improvement processing with the aid of multiple learning models, the control unit 900 can perform image quality improvement processing more effectively, wherein the multiple learning models undergo learning using training data consisting of pairs of groups that are grouped for each camera condition or for each en-face image generation range.

[0178] Note that although an example in which the selection unit 925 selects a learning model based on imaging conditions such as an imaging site or a generation range related to an OCTA image is described in the present embodiment, a configuration in which the learning model is changed based on conditions other than the above conditions may be adopted. For example, the selection unit 925 may select a learning model based on a projection method (maximum intensity projection method or average intensity projection method) when generating an OCTA image or an intensity en-face image, or whether an artifact removal process for removing artifacts caused by a vascular shadow is performed. In this case, a learning model that has been learned using respective types of training data according to the projection method and whether an artifact removal process has been performed may be prepared.

[0179] (Variant 3)

[0180] In Embodiment 2, the selection unit 925 automatically selects an appropriate learning model according to the imaging conditions or the range of generation of the en-face image, etc. In this regard, there is also a case where the operator wishes to manually select the image quality improvement processing to be applied to the image. Therefore, the selection unit 925 can select the learning model according to the operator's instruction.

[0181] Furthermore, there are also cases where the operator wishes to change the image quality improvement processing applied to the image. Therefore, the selection unit 925 may change the learning model according to the operator's instruction to change the image quality improvement processing to be applied to the image.

[0182] In the following, reference is made to Fig.11A and Fig. 11B An operation method when manually changing the image quality improvement processing to be applied to an image is described. Fig.11A and Fig. 11B Each shows an example of a report screen that switches display between images before and after image quality improvement processing. Fig.11A On the report screen 1100 shown, a tomographic image 1111 and an OCTA image 1101 to which image quality improvement processing using an automatically selected learning model has been applied are shown. Fig. 11B On the report screen 1100 shown, a tomographic image 1111 and an OCTA image 1102 to which image quality improvement processing using a learning model according to an instruction of the operator has been applied are shown. Fig.11A and Fig. 11B On the illustrated report screen 1100 , a process designation portion 1120 for changing the image quality improvement process applied to the OCTA image is shown.

[0183] Show in Fig.11A The OCTA image 1101 on the report screen 1100 shown is an OCTA image in which deep capillaries in the macular region are visualized. On the other hand, the image quality improvement processing applied to the OCTA image using the learning model automatically selected by the selection unit 925 is a processing suitable for radial papillary capillaries (RPC). Fig.11A In the OCTA image 1101 displayed on the report screen 1100 shown in the figure, the image quality improvement process applied to the OCTA image is not optimal for the blood vessels extracted in the OCTA image.

[0184] Therefore, the operator selects "deep capillaries" in the processing designation section 1120 through the input unit 260. In response to the selection instruction from the operator, the selection unit 925 changes the learning model used by the image quality improvement unit 224 to a learning model that has been learned using the OCTA images related to the deep capillaries in the macular area as training data.

[0185] The image quality improvement unit 224 uses the learning model changed by the selection unit 925 to perform image quality improvement processing on the OCTA image again. Fig. 11B As shown, the display control unit 250 causes the high-quality OCTA image 1102 newly generated by the image quality improvement unit 224 to be displayed on the display unit 270 .

[0186] Therefore, by configuring the selection unit 925 to change the learning model in response to the operator's instruction, the operator can re-designate an appropriate image quality improvement process to be performed for the same OCTA image. In addition, the designation of the image quality improvement process can be performed any number of times.

[0187] Here, an example has been shown in which the control unit 900 is configured so that the image quality improvement processing to be applied to the OCTA image can be manually changed. In this regard, the control unit 900 may also be configured so that the image quality improvement processing to be applied to the tomographic image or the intensity en-face image, etc. can be manually changed.

[0188] In addition, despite Fig.11A and Fig. 11B The report screen shown is a form in which the images before and after the image quality improvement process are switched and displayed, but the report screen may be a form in which the images before and after the image quality improvement process are displayed in a juxtaposed manner or in an overlaid manner. In addition, the form of the process designation part 1120 is not limited to Fig.11A and Fig. 11B The form shown, and can be any form that allows issuing instructions about image quality improvement processing or learning models. In addition, Fig.11A and Fig. 11B The kind of image quality improvement processing shown is one example, and other kinds of image quality improvement processing according to training data for learning a model may also be included.

[0189] In addition, similar to Modification 2, a plurality of images to which image quality improvement processing is applied may be displayed simultaneously. In this case, a configuration may also be adopted that enables specification of image quality improvement processing to be applied. An example of a report screen in this case is as follows: Fig. 12A and Fig. 12B shown.

[0190] Fig. 12A and Fig. 12B Each shows an example of a report screen that switches display between a plurality of images before and after image quality improvement processing. Fig. 12A The report screen 1200 shown shows an OCTA image 1201 before image quality improvement processing. Fig. 12BThe report screen 1200 shown in FIG. 1 shows an OCTA image 1202 to which image quality improvement processing has been applied according to the operator's instruction. Fig. 12A and Fig. 12B On the illustrated report screen 1200 , a process designation portion 1220 for changing the image quality improvement process applied to the OCTA image is shown.

[0191] In this case, the selection unit 925 selects a learning model as a learning model to be used by the image quality improvement unit 224 according to the image quality improvement process for which the instruction is issued using the process designation section 1220. The image quality improvement unit 224 performs the image quality improvement process on the plurality of OCTA images 1201 using the learning model selected by the selection unit 925. The display control unit 250 causes the generated plurality of OCTA images 1202 having high image quality to be simultaneously displayed on the report screen 1200, as shown in FIG. Fig. 12B shown.

[0192] Note that although the image quality improvement processing for OCTA images has been described above, with regard to the image quality improvement processing for tomographic images or intensity en-face images, etc., it is also possible to select and change the learning model according to the operator's instruction. Note that a plurality of images before and after the image quality improvement processing may also be displayed on the report screen in a juxtaposed manner or in an overlaid manner. In this case, it is also possible to simultaneously display a plurality of images to which the image quality improvement processing has been applied according to the operator's instruction.

[0193] (Example 3)

[0194] In Embodiments 1 and 2, the image quality improvement unit 224 automatically performs image quality improvement processing after a tomographic image or an OCTA image is captured. However, image quality improvement processing using a learning model performed by the image quality improvement unit 224 may sometimes take a long time. In addition, it takes time to generate motion contrast data by the motion contrast generation unit 222 and to generate an OCTA image by the en-face image generation unit 223. Therefore, in the case of waiting for the image quality improvement processing to be completed after the image is captured to display the image, it may take a long time after the image is captured to display the image.

[0195] In this regard, when an eye to be inspected is imaged using an OCT apparatus, in some cases, the image cannot be successfully imaged due to blinking or unconscious movement of the eye to be inspected, etc. Therefore, by allowing the operator to confirm whether the image capturing is successful at an early stage, the convenience of the OCT apparatus can be enhanced. Therefore, in Embodiment 3, the OCT apparatus is configured so that the image captured can be confirmed at an early stage by displaying an intensity en-face image or an OCTA image based on tomographic information obtained by imaging the eye to be inspected before generating and displaying a high-quality OCTA image.

[0196] In the following, reference is made to Fig.13 The OCT apparatus according to the present embodiment is described. Note that since the configuration of the OCT apparatus according to the present embodiment is similar to that of the OCT apparatus 1 according to Embodiment 1, Figure 1 Components identical to those shown in the embodiment are denoted by the same reference numerals as those in the embodiment 1, and descriptions of these components are omitted hereinafter. Hereinafter, the OCT apparatus according to this embodiment will be described centering on the differences from the OCT apparatus 1 according to the embodiment 1.

[0197] Fig.13 13 is a flowchart of a series of image processing operations according to the present embodiment. First, in step S1301 , the obtaining unit 210 obtains a plurality of items of three-dimensional tomographic information obtained by imaging the subject's eye E with the OCT imaging device 100 .

[0198] Since step S1302 is the same as step S502 according to Embodiment 1, description thereof is omitted here. When a three-dimensional tomographic image is generated in step S1302, the process proceeds to step S1303.

[0199] In step S1303, the en-face image generation unit 223 generates a frontal image (intensity en-face image) of the fundus by projecting the three-dimensional tomographic image generated in step S1302 onto a two-dimensional plane. Thereafter, in step S1304, the display control unit 250 displays the generated intensity en-face image on the display unit 270.

[0200] Since step S1305 and step S1306 are the same as steps S503 and S504 according to Embodiment 1, description of these steps is omitted here. When the OCTA image is generated in step S1306, the process proceeds to step S1307. In step S1307, the display control unit 250 causes the display unit 270 to switch from displaying the intensity en-face image to displaying the OCTA image before the image quality improvement process generated in step S1306.

[0201] In step S1308, the image quality improving unit 224 subjects the OCTA image generated in step S1306 to image quality improvement processing using the learning model, thereby generating a high-quality OCTA image, similarly to step S505 according to Embodiment 1. In step S1309, the display control unit 250 causes the display unit 270 to switch from displaying the OCTA image before the image quality improvement processing to displaying the generated high-quality OCTA image.

[0202] As described above, before the acquisition unit 210 acquires the OCTA image, the display control unit 250 according to the present embodiment causes the intensity en-face image (third image) which is a frontal image generated based on the tomographic data acquired in the depth direction of the eye to be inspected to be displayed on the display unit 270. Furthermore, immediately after the OCTA image is acquired, the display control unit 250 causes the display unit 270 to switch from displaying the intensity en-face image to displaying the OCTA image. In addition, after the image quality improvement unit 224 generates a high-quality OCTA image, the display control unit 250 causes the display unit 270 to switch from displaying the OCTA image to displaying the high-quality OCTA image.

[0203] Therefore, the operator can check the frontal image of the eye to be inspected immediately after imaging, and can immediately determine whether the imaging was successful. In addition, since the OCTA image is displayed immediately after it is generated, the operator can determine at an early stage whether a plurality of three-dimensional tomographic information for generating motion contrast data has been properly obtained.

[0204] Note that regarding a tomographic image or an intensity en-face image or the like, by displaying the tomographic image or the intensity en-face image before performing image quality improvement processing, the operator can determine at an early stage whether imaging is successful.

[0205] Although in the present embodiment, the process for generating motion contrast data (step S1305) is started after the process for displaying the intensity en-face image (step S1304), the timing of starting the process for generating motion contrast data is not limited thereto. For example, the motion contrast generation unit 222 may start the process for generating motion contrast data simultaneously with the process for generating the intensity en-face image (step S1303) or the process for displaying the intensity en-face image (step S1304). Similarly, the image quality improvement unit 224 may start the image quality improvement process (step S1308) simultaneously with the process for displaying the OCTA image (step S1307).

[0206] (Example 4)

[0207] In Embodiment 1, an example of switching and displaying OCTA images before and after image quality improvement processing is described. In contrast, in Embodiment 4, a comparison is made between images before and after image quality improvement processing.

[0208] In the following, reference is made to Fig.14 and Fig.15 The OCT apparatus according to the present embodiment is described. Note that since the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to Embodiment 1 except for the control unit, Figure 1 Components identical to those shown in the embodiment are denoted by the same reference numerals as those in the embodiment 1, and descriptions of these components are omitted hereinafter. Hereinafter, the OCT apparatus according to this embodiment will be described centering on the differences from the OCT apparatus 1 according to the embodiment 1.

[0209] Fig.14 14 is a diagram showing a schematic configuration of a control unit 1400 according to the present embodiment. Note that the components of the control unit 1400 according to the present embodiment are the same as those of the control unit 200 according to Embodiment 1, except for the image processing unit 1420 and the comparison unit 1426. Figure 2 The same components as those shown in are denoted by the same reference numerals as those in Embodiment 1, and description of these components is omitted hereinafter.

[0210] In addition to the tomographic image generation unit 221 , the motion contrast generation unit 222 , the en-face image generation unit 223 , and the image quality improvement unit 224 , a comparison unit 1426 is provided in the image processing unit 1420 of the control unit 1400 .

[0211] The comparison unit 1426 compares the image (original image) before the image quality improvement process and the image after the image quality improvement process by the image quality improvement unit 224. More specifically, the comparison unit 1426 compares the images before and after the image quality improvement process, and calculates each difference between the pixel values ​​at corresponding pixel positions in the images before and after the image quality improvement process.

[0212] Then, the comparison unit 1426 generates a colored color map image according to the size of the difference. For example, in the case where the pixel value of the image after the image quality improvement process is larger than the pixel value of the image before the image quality improvement process, a warm color (yellow to orange to red) tone is used, and in the case where the pixel value of the image after the image quality improvement process is smaller, a cool color (yellow-green to green to blue) tone is used. By using this color scheme, it can be easily recognized that the position indicated by the warm color on the color map image is the tissue that has been restored (or newly created) by the image quality improvement process. Similarly, it can be easily recognized that the position indicated by the cool color on the color map image is the noise that has been removed (or the tissue that has been erased) by the image quality improvement process.

[0213] Note that the color scheme of the color image discussed is an example. For example, the color scheme of the color image can be arbitrarily set according to a desired configuration, for example, a color scheme in which a different hue is applied according to the size of the pixel value in the image after the image quality improvement process relative to the pixel value in the image before the image quality improvement process.

[0214] The display control unit 250 may superimpose the color map image generated by the comparison unit 1426 on the image before the image quality improvement process or on the image after the image quality improvement process and display the resulting superimposed image on the display unit 270 .

[0215] Next, refer to Fig.15 A series of image processing operations according to the present embodiment are described. Note that since steps S1501 to S1505 are the same as steps S501 to S505 according to Embodiment 1, description of these steps is omitted here. When a high-quality OCTA image is generated by the image quality improvement unit 224 in step S1505, the process proceeds to step S1506.

[0216] In step S1506, the comparison unit 1426 compares the OCTA image generated in step S1504 with the high-quality OCTA image generated in step S1505 to calculate the difference between each pixel value, and generates a color map image based on the difference between each pixel value. Note that the comparison unit 1426 may perform comparison between images using other methods such as by using a ratio or correlation value of pixel values ​​between images before and after high-quality image processing, instead of the difference between pixel values ​​in the images before and after the image quality improvement process, and may generate a color map image based on the comparison result.

[0217] In step S1507, the display control unit 250 superimposes the color map image on the image before the image quality improvement process or on the image after the image quality improvement process, and displays the resulting superimposed image on the display unit 270. At this time, the display control unit 250 can set transparency for the color map to ensure that the color map image does not block the image superimposed with the color map image, and display the color map image in a superimposed manner on the target image.

[0218] In addition, in the color map image, the display control unit 250 can set the transparency to a high value at a position where the difference between the images before and after the image quality improvement process is small (the pixel value of the color map image is low), or can set the transparency so that the position where the difference is less than or equal to the predetermined value is completely transparent. By setting the transparency in this way, both the image displayed below the color map image and the color map image can be visually recognized in an advantageous manner. Note that regarding the transparency of the color map image, the comparison unit 1426 can also generate a color map image including the transparency setting.

[0219] As described above, the control unit 1400 according to the present embodiment includes the comparison unit 1426, which compares the first image and the second image subjected to the image quality improvement process. The comparison unit 1426 calculates the difference between the first image and the second image, and generates a color map image with color based on the difference. The display control unit 250 controls the display of the display unit 270 based on the comparison result obtained by the comparison unit 1426. More specifically, the display control unit 250 superimposes the color map image on the first image or the second image, and displays the resulting superimposed image on the display unit 270.

[0220] Therefore, by observing the color map image superimposed on the images before and after the image quality improvement process, the change between the images caused by the image quality improvement process can be easily confirmed. Therefore, even if a tissue that does not actually exist is visualized in the image by the image quality improvement process, or a tissue that originally existed is erased from the image by the image quality improvement process, the operator can more easily recognize such tissue and can more easily determine the authenticity of the tissue. In addition, based on the color scheme of the color map image, the operator can easily recognize whether the position is a position newly visualized by the image quality improvement process or a position erased by the image quality improvement process.

[0221] Note that the display control unit 250 can enable or disable the superimposed display of the color map image according to an instruction from the operator. The operation for turning on or off the superimposed display of the color map image can be applied to multiple images displayed on the display unit 270 at the same time. In this case, the comparison unit 1426 can generate a color map image for each of the corresponding images before and after the image quality improvement process, and the display control unit 250 can superimpose and display the color map image on the corresponding image before the image quality improvement process or the image after the image quality improvement process. In addition, the display control unit 250 can cause the image before the image quality improvement process or the image after the image quality improvement process to be displayed on the display unit 270 before displaying the color map image.

[0222] Note that although this embodiment is described by taking an OCTA image as an example, similar processing can be performed in the case of performing image quality improvement processing on a tomographic image or an intensity en-face image, etc. In addition, the comparison processing and the processing for displaying a color map according to this embodiment can also be applied to the OCT apparatus according to Embodiment 2 and Embodiment 3.

[0223] (Variant 4)

[0224] In addition, the comparison unit 1426 can compare the images before and after the image quality improvement process, and according to the comparison result of the comparison unit 1426, the display control unit 250 can display a warning on the display unit 270. Specifically, in the case where the difference between the pixel values ​​in the images before and after the image quality improvement process calculated by the comparison unit 1426 is greater than a predetermined value, the display control unit 250 displays a warning on the display unit 270. According to such a configuration, in the case where a tissue that does not actually exist is generated by the learning model in the generated high-quality image or a tissue that originally existed is erased, it is possible to draw the operator's attention to this fact. Note that the comparison between the difference and the predetermined value can be performed by the comparison unit 1426, or can be performed by the display control unit 250. In addition, instead of the difference, a statistical value such as an average value of the difference can be compared with the predetermined value.

[0225] In addition, the display control unit 250 may be configured so that, in the case where the difference between the images before and after the image quality improvement process is greater than a predetermined value, the image after the image quality improvement process is not displayed on the display unit 270. In this case, in the generated high-quality image, if a tissue that does not actually exist is generated or a tissue that originally existed is erased by the learning model, it is possible to suppress the occurrence of misdiagnosis based on the high-quality image in question. Note that the comparison between the difference and the predetermined value may be performed by the comparison unit 1426, or may be performed by the display control unit 250. In addition, instead of the difference, a statistical value such as an average value of the difference may be compared with the predetermined value.

[0226] (Example 5)

[0227] Next, refer to Fig. 20A and Fig. 20B The image processing device (control unit 200) according to Embodiment 5 is described. In this embodiment, an example is described in which the display control unit 250 displays the processing result of the image quality improvement unit 224 on the display unit 270. Note that although the image quality improvement unit 224 is displayed using the display control unit 250, the display control unit 270 is displayed. Fig. 20A and Fig. 20B This embodiment is described, but the display screen is not limited to Fig. 20A and Fig. 20B The example shown. As in the case of subsequent observation, the image quality improvement process can also be similarly applied to a display screen that displays multiple images obtained at different dates and times side by side. In addition, as in the case of the imaging confirmation screen, the image quality improvement process can also be similarly applied to a display screen where the examiner confirms whether the imaging is successful immediately after imaging. The display control unit 250 can cause the display unit 270 to display multiple high-quality images generated by the image quality improvement unit 224 or low-quality images that have not been improved in image quality. Therefore, the display control unit 250 can output low-quality images and high-quality images respectively according to the instructions of the examiner.

[0228] In the following, reference is made to Fig. 20A and Fig. 20BAn example of the interface 3400 discussed is described. Reference numeral 3400 denotes the entire screen, reference numeral 3401 denotes a "patient" label, reference numeral 3402 denotes a "camera" label, reference numeral 3403 denotes a "report" label, and reference numeral 3404 denotes a "settings" label. In addition, the diagonal line in the "report" label 3403 indicates the activation state of the report screen. In the present embodiment, an example of displaying a report screen will be described. Reference numeral Im3405 denotes an SLO image, and reference numeral Im3406 denotes an image in which an OCTA en-face image indicated by reference numeral Im3407 is displayed in a superimposed manner on the SLO image Im3405. Here, the term "SLO image" refers to a frontal image of the fundus obtained by an SLO (scanning laser ophthalmoscope) optical system (not shown). Reference numerals Im3407 and Im3408 each denote an OCTA en-face image, reference numeral Im3409 denotes an intensity en-face image, and reference numerals Im3411 and Im3412 each denote a tomographic image. Reference numerals 3413 and 3414 denote boundary lines of upper and lower ranges of the OCTA en-face images indicated by Im3407 and Im3408, respectively, which are displayed in a superimposed manner on the corresponding tomographic images. Button 3420 is a button for designating execution of image quality improvement processing. Naturally, as described later, button 3420 may be a button for inputting an instruction to display a high-quality image.

[0229] In the present embodiment, execution of the image quality improvement process is performed when the button 3420 is designated, or whether to perform the image quality improvement process is determined based on information stored (saved) in the database. First, an example of switching between display of a high-quality image and display of a low-quality image by designating the button 3420 according to an instruction from the examiner will be described. Note that an OCTA en-face image is described as a target image for the image quality improvement process.

[0230] When the inspector performs an operation to designate the "Report" tab 3403 and thereby transition to the report screen, low-quality OCTA en-face images Im3407 and Im3408 are displayed. Thereafter, when the inspector performs an operation to designate button 3420, the image quality improvement unit 224 performs image quality improvement processing on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement processing is completed, the display control unit 250 displays the high-quality image generated by the image quality improvement unit 224 on the report screen. Note that since the image indicated by reference numeral Im3406 is an image obtained by displaying image Im3407 on SLO image Im3405 in a superimposed manner, image Im3406 is also an image that has undergone image quality improvement processing. Then, the display of button 3420 is changed to an activated state to provide a display that can be understood as performing image quality improvement processing.

[0231] In this case, the execution of the processing by the image quality improvement unit 224 does not have to be limited to the timing when the examiner performs the operation of the designated button 3420. Since the types of OCTA en-face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, the image quality improvement processing can be performed when the report screen is switched to. Subsequently, at the timing of pressing the button 3420, the display control unit 250 can display the high-quality image on the report screen. In addition, the number of types of images to be subjected to image quality improvement processing in response to an instruction from the examiner or when switching to the report screen does not have to be two. Such a construction may also be adopted to process images with a high probability of being displayed, for example, images such as Fig.19A and Fig.19B In this case, the image obtained by performing the image quality improvement process may be temporarily stored in a memory or may be stored in a database.

[0232] Next, a case where an image quality improvement process is performed based on information stored (saved) in a database is described. In the case where the state of execution of the image quality improvement process to be performed is stored in the database, when the display is switched to the report screen, a high-quality image obtained by performing the image quality improvement process is displayed by default. In addition, such a construction can be adopted: the default button 3420 is displayed in an activated state, so that the inspector can know that a high-quality image obtained by performing the image quality improvement process is being displayed. If the inspector wishes to display a low-quality image in a state before the image quality improvement process, the inspector can display the low-quality image by performing an operation of the designated button 3420 to thereby release the activated state. If the inspector wishes to return to a high-quality image, the inspector designates the button 3420.

[0233] It is assumed that for all data stored in the database, and for each category of data such as each group of imaging data (for each examination), it is possible to collectively specify whether to perform image quality improvement processing on the data stored in the database. For example, in the case where a state in which image quality improvement processing is to be performed on the entire database has been stored, if the examiner stores a state in which image quality improvement processing is not performed on a single item (single examination) of the imaging data, the imaging data will be displayed in a state in which image quality improvement processing has not yet been performed on the imaging data when the relevant imaging data is displayed next time. A user interface (not shown) (e.g., a "save" button) can be used to store a state in which image quality improvement processing has been performed on each item of the imaging data (for each examination). In addition, when switching to other imaging data (other examinations) or other patient data (e.g., changing to a display screen other than a report screen according to an instruction from the examiner), the state in which image quality improvement processing is to be performed can be stored based on the display state (e.g., the state of button 3420). In this way, in a case where it has not been specified whether to perform image quality improvement processing on a per-camera data unit (inspection unit) basis, processing can be performed based on information specified for the entire database, whereas in a case where it has been specified to perform image quality improvement processing on a per-camera data unit (inspection unit) basis, processing can be performed individually based on the information in question.

[0234] Although an example in which images Im3407 and Im3408 are displayed as OCTA en-face images is shown in the present embodiment, the OCTA en-face images to be displayed may be changed according to the inspector's designation. Therefore, a description will now be given of changing the image when execution of the image quality improvement process has been designated (button 3420 is in an activated state).

[0235] The image is changed using a user interface (not shown) (e.g., a combo box). For example, when the examiner changes the type of image from a surface layer image to a choroidal vascular network image, the image quality improvement unit 224 performs image quality improvement processing on the choroidal vascular network image, and the display control unit 250 displays the high-quality image generated by the image quality improvement unit 224 on the report screen. In other words, in response to an instruction from the examiner, the display control unit 250 can change the display of a high-quality image of a first depth range to a display of a high-quality image of a second depth range that is at least partially different from the first depth range. At this time, by changing the first depth range to the second depth range in response to an instruction from the examiner, the display control unit 250 can change the display of a high-quality image of the first depth range to a display of a high-quality image of the second depth range. Please note that, in the case where a high-quality image has been generated for an image with a high probability of being displayed when transitioning to the report screen as described above, the display control unit 250 can display the high-quality image that has been generated.

[0236] Note that the method for changing the image kind is not limited to the above method, and an OCTA en-face image with a different depth range set can also be generated by changing the layer used as a reference and the offset value. In this case, when the layer used as a reference or the offset value is changed, the image quality improvement unit 224 performs image quality improvement processing for any OCTA en-face image, and the display control unit 250 displays a high-quality image on the report screen. The change of the layer used as a reference or the offset value can be made using a user interface (not shown) (for example, a combo box or a text box). In addition, the range for generating the OCTA en-face image can be changed by dragging any one of the boundary lines 3413 and 3414 displayed on the tomographic images Im3411 and Im3412 in an overlaid manner (moving the layer boundary).

[0237] In the case where the boundary line is changed by dragging, the execution command for the image quality improvement process is continuously issued. Therefore, the image quality improvement unit 224 may always process for the execution command, or may be configured to execute the process after the layer boundary is changed by dragging. Alternatively, although the execution of the image quality improvement process is continuously issued, the image quality improvement unit 224 may be configured to cancel the previous command at the time point when the next command arrives, and execute the latest command.

[0238] Note that in some cases, the image quality improvement process takes a relatively long time. Therefore, even when the command is executed at any timing in the above timing, it may take a relatively long time until a high-quality image is displayed. Therefore, during the period from when the depth range for generating the OCTA en-face image is set in response to the instruction from the examiner to when the high-quality image is displayed, the OCTA en-face image (low-quality image) corresponding to the set depth range can be displayed. In other words, the following configuration can be adopted: when the above-mentioned depth range is set, the OCTA en-face image (low-quality image) corresponding to the set depth range is displayed, and when the image quality improvement process is completed, the display of the relevant OCTA en-face image (low-quality image) is changed to the display of the high-quality image. In addition, information indicating that the image quality improvement process is being performed can be displayed during the period from when the above-mentioned depth range is set to when the high-quality image is displayed. Note that the foregoing can be applied not only to the case where the state is assumed to be a state in which the execution of the image quality improvement process has been specified (button 3420 is in an activated state), but can also be applied to, for example, the period from when the execution of the image quality improvement process is instructed according to the instruction of the examiner until the high-quality image is displayed.

[0239] Although an example is shown in the present embodiment in which different layers are displayed as images Im3407 and Im3408 as OCTA en-face images, and a low-quality image and a high-quality image are displayed by switching between them, the present invention is not limited thereto. For example, a low-quality OCTA en-face image as image Im3407 and a high-quality OCTA en-face image as image Im3408 may be displayed side by side. In the case of displaying images by switching between them, since the images are switched at the same place, it is easy to compare the changed portions, and in the case of displaying the images side by side, since the images can be displayed at the same time, it is easy to compare the entire images.

[0240] Next, Fig. 20A and Fig. 20B The execution of the image quality improvement process in the case of screen transition will be described. Fig. 20B It is displayed in enlarged mode Fig. 20A An example of a screen of the OCTA en-face image Im3407 is shown. Fig. 20B In, similar to Fig. 20A , display button 3420. For example, by double-clicking the OCTA en-face image Im3407, specifying Fig. 20A The screen shown to Fig. 20B The screen transition of the screen shown in FIG. 34 is shown, and the screen transition from the screen shown in FIG. 34 is specified by clicking the "Close" button 3430. Fig. 20B The screen shown to Fig. 20A Note that regarding screen transition, the method for transitioning from one screen to another is not limited to the method described here, and a user interface (not shown) may also be used.

[0241] When the image quality improvement process has been specified to be executed at the time of screen transition (button 3420 is in the activated state), this state is maintained even when screen transition occurs. Fig. 20B The screen shown changes to Fig. 20A In the case where a high-quality image is displayed on the screen shown in the figure, the high-quality image is also displayed on the Fig. 20B In addition, button 3420 is activated. The same applies to Fig. 20B The screen changes to Fig. 20A The screen shown in the figure is Fig. 20B On the screen shown, the display can also be switched to a low-quality image by specifying button 3420.

[0242] Regarding the screen transition, the screen transition is not limited to the screen described here, and as long as the transition is made to a screen displaying the same camera data, for example, a display screen for subsequent observation or a display screen for a panoramic image, the transition can be made while maintaining the display state of the high-quality image. In other words, on the display screen after the transition, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed. For example, if the button 3420 is in an activated state on the display screen before the transition, a high-quality image is displayed on the display screen after the transition. In addition, for example, if the activation state of the button 3420 is released on the display screen before the transition, a low-quality image is displayed on the display screen after the transition. Note that the following configuration may be adopted: if the button 3420 is in an activated state on the display screen for subsequent observation, a plurality of images displayed side by side on the display screen for subsequent observation obtained at different dates and times (different examination dates) are switched to high-quality images. In other words, the following configuration may be adopted: if the button 3420 is in an activated state on the display screen for subsequent observation, switching to a high-quality image is performed collectively for a plurality of images obtained at different dates and times.

[0243] exist Fig.18 An example of a display screen for subsequent observation is shown in FIG. When the tab 3801 is selected in response to an instruction from the examiner, the following is displayed: Fig.18The display screen for subsequent observation shown in FIG. 38 is a diagram showing a display screen for subsequent observation. At this time, the depth range of the en-face image can be changed by the examiner performing an operation of selecting from a set of predefined depth ranges (3802 and 3803) displayed in a list box. For example, surface capillaries are selected in list box 3802, and deep capillaries are selected in list box 3803. The analysis results of the en-face image of the surface capillaries are displayed in the display area on the upper side, and the analysis results of the en-face image of the deep capillaries are displayed in the display area on the lower side. In other words, when the depth range is selected, a plurality of images obtained at different dates and times are collectively changed to display in parallel the analysis results of a plurality of en-face images within the selected depth range.

[0244] At this time, if the display of the analysis results is placed in an unselected state, the display can be collectively changed to display the analysis results of multiple en-face images obtained at different dates and times in parallel. In addition, if button 3420 is designated according to an instruction from the examiner, the display of multiple en-face images is collectively changed to display of multiple high-quality images.

[0245] In addition, when the display of the analysis results is in the selected state, if the button 3420 is specified according to the instruction from the examiner, the display of the analysis results of the plurality of en-face images is collectively changed to the display of the analysis results of the plurality of high-quality images. Here, the display of the analysis results can be performed so that the analysis results are displayed on the image in an overlaid manner with any transparency. At this time, the change to the display of the analysis results can be, for example, a change to a state in which the analysis results are overlaid on the image being displayed with any transparency. In addition, the change to the display of the analysis results can be, for example, a change to the display of each analysis result and image (e.g., a two-dimensional graph) obtained by subjecting each analysis result and image to fusion processing with any transparency.

[0246] In addition, such layer boundaries and offset positions for specifying the depth range can be collectively changed from user interfaces such as those indicated by reference numerals 3805 and 3806, respectively. Note that the depth ranges of a plurality of en-face images acquired at different dates and times can be collectively changed by displaying a tomographic image together with the tomographic image and moving the layer boundary data superimposed on the tomographic image in accordance with an instruction from the examiner. At this time, a plurality of tomographic images acquired at different dates and times can be displayed side by side, and when the above-mentioned movement is performed on one tomographic image, the layer boundary data can be similarly moved on the other tomographic images.

[0247] For example, the image projection method and whether or not to perform projection artifact removal processing can be changed by selection from a user interface such as a context menu.

[0248] In addition, the selection button 3807 can be selected to display a selection screen, and an image selected from the image list displayed on the selection screen can be displayed. Fig.18 The arrow 3804 displayed in the upper portion of the illustrated screen is a mark indicating the currently selected examination, and the reference examination (baseline) is the examination selected in subsequent imaging ( Fig.18 Naturally, a mark indicating the reference check may be displayed on the display unit.

[0249] Furthermore, in the case where the "show difference" check box 3808 is specified, the measurement value distribution (graph or sector graph) about the reference image is displayed on the reference image. In addition, in this case, in the area corresponding to the inspection date other than the inspection date of the reference image, a differential measurement value graph is displayed, which shows the difference between the measurement value distribution calculated for the reference image and the measurement distribution value calculated for the image displaying the relevant area. As the measurement result, a trend curve graph (a curve graph of the measurement values ​​of the image on each inspection date obtained by measuring the change over time) can be displayed on the report screen. In other words, time series data (for example, a time series graph) of multiple analysis results corresponding to multiple images obtained at different dates and times can be displayed. At this time, with respect to the analysis results related to the date and time other than the multiple dates and times corresponding to the multiple images displayed, the analysis results can also be displayed as time series data in a state where the analysis results can be distinguished from the multiple analysis results corresponding to the multiple images being displayed (for example, the color of each point on the time series graph is different depending on whether the corresponding image is displayed). In addition, the regression line (curve) of the trend curve graph and the corresponding mathematical formula can be displayed on the report screen.

[0250] Although a description has been given in the present embodiment regarding an OCTA en-face image, the present invention is not limited thereto. The image related to the processing for displaying an image, image quality improvement, and image analysis, etc. according to the present embodiment may be an intensity en-face image. In addition, the type of image is not limited to an en-face image, and may be a different type of image such as a tomographic image, an SLO image, a fundus image, or a fluorescent fundus image. In this case, the user interface for performing the image quality improvement processing may be a user interface for instructing the execution of the image quality improvement processing for a plurality of images of different types, or may be a user interface for selecting any image from a plurality of images of different types and instructing the execution of the image quality improvement processing.

[0251] According to the aforementioned configuration, the display control unit 250 can display the image processed by the image quality improvement unit 224 according to the present embodiment on the display unit 270. At this time, as described above, in a state where at least one condition is selected among a plurality of conditions related to the display of a high-quality image, the display of an analysis result, and the depth range of a front image to be displayed, even if the display screen is changed to another display screen, the selected state can be maintained.

[0252] In addition, as described above, in the case where the state of at least one condition among the multiple conditions is selected, even if the other conditions are changed to the selected state, the state of selecting at least one condition can be maintained. For example, in the case where the display of the analysis result is in the selected state, in response to an instruction from the inspector (for example, when the button 3420 is designated), the display control unit 250 can change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image. In addition, in the case where the display of the analysis result is in the selected state, in response to an instruction from the inspector (for example, when the designation of the button 3420 is released), the display control unit 250 can change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image.

[0253] Moreover, in the case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the designation of the display of the analysis result is released), the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the low-quality image. In addition, in the case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the display of the analysis result is designated), the display control unit 250 may change the display of the low-quality image to the display of the analysis result of the low-quality image. In addition, in the case where the display of the high-quality image is in a selected state, in response to an instruction from the inspector (for example, when the designation of the display of the analysis result is released), the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the high-quality image. In addition, in the case where the display of the high-quality image is in a selected state, in response to an instruction from the inspector (for example, when the display of the analysis result is designated), the display control unit 250 may change the display of the high-quality image to the display of the analysis result of the high-quality image.

[0254] Furthermore, let us consider a case where the display of the high-quality image is in an unselected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second analysis result is specified), the display control unit 250 may change the display of the first analysis result of the low-quality image to the display of the second analysis result of the low-quality image. Furthermore, let us consider a case where the display of the high-quality image is in a selected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the examiner (for example, when the display of the second analysis result is specified), the display control unit 250 may change the display of the first analysis result of the high-quality image to the display of the second analysis result of the high-quality image.

[0255] Note that the following construction may be adopted: on the display screen for subsequent observation, as described above, for a plurality of images obtained at different dates and times, changes in these displays are collectively reflected. Here, the display of the analysis results may be performed so that the analysis results are displayed on the image in an overlaid manner with any transparency. At this time, the display changed to the analysis results may be, for example, changed to a state in which the analysis results are overlaid on the image being displayed with any transparency. In addition, the display changed to the analysis results may be, for example, changed to the display of the individual analysis results and images (e.g., a two-dimensional graph) obtained by subjecting the individual analysis results and images to fusion processing with any transparency.

[0256] (Variant 5)

[0257] In the various embodiments and modifications described above, the display control unit 224 can cause the display unit 270 to display an image selected from the high-quality image generated by the image quality improvement unit 404 and the input image according to an instruction from the inspector. In addition, in response to an instruction from the inspector, the display control unit 250 can switch the image displayed on the display unit 270 from the camera image (input image) to the high-quality image. In other words, the display control unit 250 can change the display of the low-quality image to the display of the high-quality image in response to an instruction from the inspector. In addition, the display control unit 250 can change the display of the high-quality image to the display of the low-quality image in response to an instruction from the inspector.

[0258] In addition, the image quality improvement unit 224 can start the image quality improvement process (input the image into the image quality improvement engine) by the image quality improvement engine (learning model for improving image quality) in response to the instruction from the examiner, and the display control unit 224 can cause the display unit 270 to display the high-quality image generated by the image quality improvement unit 224. In contrast, when the imaging device (OCT imaging unit 100) captures the input image, the image quality improvement engine can automatically generate a high-quality image based on the input image, and the display control unit 405 can cause the display unit 270 to display the high-quality image in response to the instruction from the examiner. Here, the term "image quality improvement engine" includes the learning model that performs the above-mentioned image quality improvement process.

[0259] Note that these processing operations can also be performed similarly for the output of the analysis results. In other words, the display control unit 250 can change the display of the analysis results of the low-quality image to the display of the analysis results of the high-quality image in response to the instruction from the inspector. In addition, the display control unit 250 can change the display of the analysis results of the high-quality image to the display of the analysis results of the low-quality image in response to the instruction from the inspector. Naturally, in response to the instruction from the inspector, the display control unit 250 can change the display of the analysis results of the low-quality image to the display of the low-quality image. In addition, in response to the instruction from the inspector, the display control unit 250 can change the display of the low-quality image to the display of the analysis results of the low-quality image. Moreover, the display control unit 250 can change the display of the analysis results of the high-quality image to the display of the high-quality image in response to the instruction from the inspector. In addition, in response to the instruction from the inspector, the display control unit 250 can change the display of the high-quality image to the display of the analysis results of the high-quality image.

[0260] In addition, the display control unit 250 can change the display of the analysis results of the low-quality image to the display of the different types of analysis results of the low-quality image in response to the instruction from the examiner. In addition, the display control unit 250 can change the display of the analysis results of the high-quality image to the display of the different types of analysis results of the high-quality image in response to the instruction from the examiner.

[0261] In this case, the display of the analysis result of the high-quality image can be performed so that the analysis result of the high-quality image is displayed on the high-quality image in an overlaid manner with any transparency. In addition, the display of the analysis result of the low-quality image can be performed so that the analysis result of the low-quality image is displayed on the low-quality image in an overlaid manner with any transparency. At this time, the change to the display of the analysis result can be, for example, a change to a state in which the analysis result is overlaid on the image being displayed with any transparency. In addition, the change to the display of the analysis result can be, for example, a change to the display of an image (for example, a two-dimensional graph) obtained by subjecting the analysis result and the image to a fusion process with any transparency.

[0262] (Variant 6)

[0263] The analysis results such as the thickness of the desired layer or the density of various blood vessels can be displayed on the report screen described in the various embodiments and modifications described above. In addition, the parameter value (distribution) related to the site of interest can be displayed as the analysis result, and the site of interest includes at least one of the optic nerve head, the macular area, the blood vessel area, the nerve bundle, the vitreous area, the macular area, the choroid area, the scleral area, the lamina cribrosa area, the retinal layer boundary, the retinal layer boundary edge, the photoreceptor cell, the blood cell, the blood vessel wall, the inner wall boundary of the blood vessel, the outer boundary of the blood vessel, the ganglion cell, the corneal area, the horn area, and the Schlemm's canal. At this time, for example, it is possible to display the accurate analysis result by analyzing the medical image after various artifact removal processes. Note that the artifact may be, for example, a false image area caused by light absorption of the blood vessel area, etc., a projection artifact, or a band-shaped artifact in the front image that appears in the main scanning direction of the measurement light due to the state of the subject's eye (movement or blinking, etc.). In addition, the artifact may be of any kind as long as it is, for example, a photographing failure area that appears randomly on the medical image of a predetermined part of the subject at each time of photographing. In addition, the value (distribution) of the parameter related to the area (imaging failure area) including at least one of the types of the above-mentioned artifacts can be displayed as the analysis result. In addition, the value (distribution) of the parameter related to the area including at least one abnormal part such as drusen, new blood vessel parts, leukocytes (hard exudate), pseudoacne, etc. can be displayed as the analysis result.

[0264] The analysis result can be displayed using an analysis graph or using sectors indicating statistical values ​​corresponding to each divided area, etc. Note that the analysis result can be generated using a learning model (analysis result generation engine, or a learning model for generating analysis results) obtained by learning the analysis result of a medical image as training data. At this time, the learning model can be a model obtained by learning using the following training data: training data including medical images and analysis results of the medical images, or training data including medical images and analysis results of medical images of different types from the relevant medical images, etc. In addition, the learning model can be a model obtained by learning using training data including input data, in which a plurality of medical images of different types of predetermined parts such as intensity en-face images and motion contrast en-face images are taken as a set. Here, the intensity en-face image corresponds to the intensity en-face image, and the motion contrast en-face image corresponds to the OCTA en-face image. In addition, the following configuration can be adopted: so that the analysis result obtained using a high-quality image generated by a learning model for improving image quality is displayed.

[0265] In addition, the input data included in the training data may be a high-quality image generated by a learning model for improving image quality, or may be a set consisting of a low-quality image and a high-quality image. In addition, the training data may be, for example, data obtained by annotating input data, for which at least one of information including an analysis value (e.g., an average value or a median value) obtained by analyzing an analysis area, a table including analysis values, an analysis graph, and the position of an analysis area such as a sector in an image is used as correct answer data (for supervised learning). Note that the following construction may be adopted: an analysis result obtained by a learning model for analysis result generation is displayed in response to an instruction from an inspector.

[0266] Furthermore, in the various embodiments and modifications described above, various diagnostic results, such as results related to glaucoma or age-related macular degeneration, can be displayed on the report screen. At this time, for example, an accurate diagnostic result can be displayed by analyzing a medical image that has been subjected to various artifact removal processes as described above. Furthermore, in the diagnostic result, the position of a specified abnormal part, etc. can be displayed on the image, and the state of the abnormal part, etc. can be displayed using characters, etc. Furthermore, a classification result (e.g., Curtin's classification) for the abnormal part can be displayed as a diagnostic result.

[0267] Note that the diagnosis result may be a result generated using a learning model (a diagnosis result generation engine, or a learning model for generating a diagnosis result) obtained by learning using the diagnosis result of a medical image as training data. In addition, the learning model may be a model obtained by learning using the following training data: training data including a medical image and the diagnosis result of the medical image, or training data including a medical image and the diagnosis result of a medical image of a type different from that of the relevant medical image, etc. In addition, a construction may be adopted so as to display a diagnosis result obtained using a high-quality image generated by a learning model for improving image quality.

[0268] In addition, the input data included in the training data may be a high-quality image generated by a learning model for improving image quality, or may be a set consisting of a low-quality image and a high-quality image. In addition, the training data may be, for example, data obtained by annotating input data, for which information including at least one of the diagnosis, the type or state (degree) of the lesion (abnormal part), the position of the lesion in the image, the position of the lesion relative to the area of ​​interest, the findings (explanation of the findings, etc.), the basis for the diagnosis (affirmative medical support information, etc.), and the basis for the negative diagnosis (negative medical support information) is used as the correct answer data (for supervised learning). Note that the following construction may be adopted: the diagnosis result obtained by the learning model for generating the diagnosis result is displayed in response to an instruction from the examiner.

[0269] In addition, in the various embodiments and variations described above, object recognition results (object detection results) or segmentation results about the focus area, artifacts, abnormal areas, etc. as described above can be displayed on the report screen. At this time, for example, a rectangular frame or the like can be displayed superimposed around the object on the image. In addition, for example, a color or the like can be superimposed on the object on the image and displayed. Please note that the object recognition result or the segmentation result can be a result generated using a learning model obtained by learning using training data, in which information indicating object recognition or segmentation is annotated as correct answer data on the medical image. Please note that the above-mentioned analysis result generation or diagnosis result generation can be achieved by utilizing the above-mentioned object recognition result or segmentation result. For example, a process for generating an analysis result or for generating a diagnosis result can be performed for the focus area obtained by the object recognition process or the segmentation process.

[0270] The above-mentioned learning model may be a learning model obtained by learning using training data including input data, in which a plurality of medical images of different kinds as images of a predetermined part of a subject are taken as a set. At this time, for example, it is conceivable to take a motion contrast positive image and an intensity positive image (or an intensity tomographic image) of the fundus as input data of the set as input data included in the training data. Furthermore, for example, it is conceivable to take a tomographic image (B-scan image) and a color fundus image (or a fluorescent fundus image) of the fundus as input data of the set as input data included in the training data. In addition, the plurality of medical images of different kinds may be of any kind as long as the medical images are obtained by different modalities, different optical systems, different principles, etc.

[0271] In addition, the above-mentioned learning model may be a learning model obtained by learning using training data including input data, in which a plurality of medical images of different parts of the subject are taken as a set. In this case, for example, it is conceivable to take a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior segment of the eye as input data of the set as input data included in the training data. In addition, for example, it is also conceivable to take a three-dimensional OCT image (three-dimensional tomographic image) of the fundus macula and a tomographic image obtained by circular scanning (or raster scanning) of the fundus optic nerve head as input data of the set as input data included in the training data.

[0272] Note that the input data included in the training data may be a plurality of medical images of different parts and different kinds of the subject. At this time, for example, it is conceivable that the input data in which the tomographic image of the anterior segment of the eye and the color fundus image are taken as a set is taken as the input data included in the training data. In addition, the above-mentioned learning model may be a learning model obtained by learning using training data including input data in which a plurality of medical images of different camera angles as images of a predetermined part of the subject are taken as a set. In addition, the input data included in the training data may be data obtained by joining together a plurality of medical images, which are obtained, for example, by temporally dividing the predetermined part into a plurality of regions as in the case of a panoramic image. At this time, by using a wide-angle image such as a panoramic image as training data, it is possible to acquire the feature value of the image with high accuracy due to the fact that the amount of information is greater than that in the case of a narrow-angle image, and thus the results of each processing can be enhanced. In addition, the input data included in the training data may be input data in which a plurality of medical images of a predetermined part of the subject obtained at different dates and times are taken as a set.

[0273] In addition, the display screen on which at least one of the above-mentioned analysis results, diagnosis results, object recognition results, and segmentation results is to be displayed is not limited to the report screen. Such a display screen may be, for example, at least one of a camera confirmation screen, a display screen for subsequent observation, and a preview screen for making various adjustments before camera shooting (a display screen on which various real-time motion images are displayed), etc. For example, by displaying the aforementioned at least one result obtained using the above-mentioned learning model on the camera confirmation screen, the examiner can check the accurate result even immediately after the camera shooting. In addition, changing the display between the above-mentioned low-quality images and high-quality images may be, for example, changing the display between the analysis results of the low-quality images and the analysis results of the high-quality images.

[0274] The above-mentioned various learning models can be obtained by machine learning using training data. For example, deep learning consisting of a multi-level neural network is a kind of machine learning. In addition, for example, a convolutional neural network (CNN) can be used for at least a part of a multi-level neural network. In addition, techniques related to autoencoders can be used for at least a part of a multi-level neural network. In addition, techniques related to back propagation (error back propagation method) can be used for learning. However, this machine learning is not limited to deep learning, and any model can be used as long as the model itself can extract (represent) feature values ​​of training data such as images through learning.

[0275] Furthermore, the image quality improvement engine (learning model for improving image quality) may be a learning model obtained by incremental learning using training data including at least one high-quality image generated by the image quality improvement engine. At this time, a configuration may be adopted in which it is possible to select whether to use a high-quality image as training data for incremental learning to be performed by an instruction from the inspector.

[0276] (Variant 7)

[0277] The following structure can be adopted: on the preview screen in the various embodiments and modifications described above, the above-mentioned learning model is used for at least one frame of the real-time motion image. At this time, it can also be configured as follows: when a plurality of real-time motion images of different parts or different types are displayed on the preview screen, the learning model corresponding to each real-time motion image is used. In this way, for example, since the processing time can be shortened even for real-time motion images, the examiner can obtain high-precision information before starting to shoot. Therefore, for example, since troubles such as re-shooting can be reduced, the accuracy and efficiency of diagnosis can be improved. Note that the plurality of real-time motion images may include, for example, motion images of the anterior segment of the eye for alignment in the XYZ direction, and frontal motion images of the fundus for focusing of the fundus observation optical system or OCT focusing. In addition, the plurality of real-time motion images may also include, for example, tomographic motion images of the fundus for coherence gate adjustment in OCT (adjustment of the optical path length difference between the measurement optical path length and the reference optical path length), etc.

[0278] In addition, the motion image to which the above-mentioned learning model can be applied is not limited to the real-time motion image, and, for example, the motion image can be a motion image stored (saved) in a storage unit. At this time, for example, a motion image obtained by aligning at least one frame of a tomographic motion image of the fundus stored (saved) in the storage unit can be displayed on the display screen. For example, in the case where it is desired to observe the vitreous body properly, first, the frame can be selected based on conditions such as that the vitreous body exists in the reference frame as much as possible. At this time, each frame is a tomographic image (B-scan image) in the XZ direction. Subsequently, a motion image in which other frames have been aligned in the XZ direction with respect to the selected reference frame can be displayed on the display screen. At this time, for example, a structure can be adopted so that high-quality images (high-quality image frames) generated sequentially for at least one frame of the motion image by a learning model for improving image quality are continuously displayed.

[0279] Note that as the above-mentioned method for aligning between frames, regarding the method for aligning in the X direction and the method for aligning in the Z direction (depth direction), the same method may be applied, or the methods applied may be different. In addition, alignment in the same direction may be performed multiple times by different methods. For example, coarse alignment may be performed, and thereafter fine alignment may be performed. In addition, the alignment method includes, for example, (coarse Z direction) alignment using the retinal layer boundary obtained by subjecting the tomographic image (B-scan image) to segmentation processing, (fine X direction or Z direction) alignment using correlation information (similarity) between a plurality of regions obtained by segmenting the tomographic image and the reference image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), and (X direction) alignment using a two-dimensional frontal image. In addition, a configuration may be adopted so that fine alignment is performed in sub-pixel units after coarse alignment is performed in pixel units.

[0280] In this case, it is possible that the imaging target such as the retina of the eye to be inspected cannot be successfully imaged during various adjustments. Therefore, since there is a large difference between the medical image input to the learning model and the medical image used as training data, it is possible that a high-quality image cannot be accurately obtained. Therefore, the following structure can also be adopted: when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B-scan) exceeds a threshold value, the display of high-quality motion images (continuous display of high-image quality frames) is automatically started. In addition, the following structure can be adopted: when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B-scan) exceeds a threshold value, the image quality improvement button is changed to a state (activated state) in which the button can be selected by the examiner.

[0281] In addition, the following configuration may be adopted: different learning models for improving image quality are prepared for each imaging mode that is different in scanning patterns, etc., and a learning model for improving image quality corresponding to the selected imaging mode is selected. In addition, a learning model for improving image quality obtained by learning using training data including various medical images obtained in different imaging modes may be used.

[0282] (Variant 8)

[0283] In the various embodiments and variations described above, in the case where a learning model is undergoing incremental learning, it is possible that it will be difficult to output (infer / predict) using the learning model itself that is undergoing incremental learning. Therefore, it is possible to prohibit the input of medical images into the learning model that is undergoing incremental learning. In addition, a learning model that is the same as the learning model that is undergoing incremental learning is prepared as another auxiliary learning model. At this time, the following construction may also be adopted: it is possible to perform the input of medical images into the auxiliary learning model that is undergoing incremental learning. Subsequently, after the incremental learning is completed, the learning model that has undergone additional learning is evaluated, and if there is no problem, it is sufficient to switch from the auxiliary learning model to the learning model that has undergone additional learning. In addition, the following construction may be adopted: if there is a problem, the auxiliary learning model is used.

[0284] In addition, the following configuration may be adopted: so that a learning model obtained by learning for each imaging part can be selectively utilized. Specifically, a plurality of learning models may be prepared, including a first learning model obtained using training data including a first imaging part (lung, eye to be inspected, etc.), and a second learning model obtained using training data including a second imaging part, which is different from the first imaging part. In addition, the control unit 200 may have a selection unit for selecting any one of the plurality of learning models. At this time, the control unit 200 may have a control unit for performing incremental learning for the selected learning model. In response to an instruction from the examiner, the control unit may retrieve data forming a pair of an imaging part corresponding to the selected learning model and an image obtained by imaging the relevant imaging part, and perform learning using the retrieved and obtained data as training data as incremental learning for the selected learning model. Note that the imaging part corresponding to the selected learning model may be a part obtained based on the title information of the data, or a part manually input by the examiner. In addition, data retrieval may be performed, for example, from a server of an external facility such as a hospital or a laboratory via a network. In this way, by using images obtained by imaging an imaging portion corresponding to a learning model, incremental learning can be efficiently performed for each imaging portion.

[0285] Note that the selection unit and the control unit may be constituted by a software module executed by a processor such as a CPU or an MPU of the control unit 200. Furthermore, the selection unit and the control unit may be constituted by a circuit having a specific function such as an ASIC or by an independent device or the like.

[0286] In addition, when training data for incremental learning is obtained from a server of an external facility such as a hospital or a laboratory through a network, it is desirable to reduce the reduction in reliability due to tampering or system failure during incremental learning, etc. Therefore, the correctness of the training data for incremental learning can be detected by confirming consistency through a digital signature or hash. In this way, the training data for incremental learning can be protected. At this time, in the event that the correctness of the training data for incremental learning cannot be detected as a result of confirming consistency through a digital signature or hash, a warning to that fact is given, and the training data in question is not used for incremental learning. Note that the server can be any form of server such as a cloud server, a fog server, or an edge server, regardless of where it is installed.

[0287] (Variant 9)

[0288] In the various embodiments and variations described above, in addition to manual instructions (for example, instructions using a user interface, etc.), the instructions from the inspector may also be voice instructions, etc. At this time, for example, a machine learning model including a voice recognition model obtained through machine learning (a voice recognition engine or a learning model for voice recognition) may be used. In addition, the manual instructions may be instructions input through characters using a keyboard, a touch panel, etc. At this time, for example, a machine learning model including a character recognition model obtained through machine learning (a character recognition engine, a learning model for character recognition) may be used. In addition, the instructions from the inspector may be instructions through gestures, etc. At this time, a machine learning model including a gesture recognition model obtained through machine learning (a gesture recognition engine, a learning model for gesture recognition) may be used.

[0289] In addition, the instruction from the examiner may be a result of detecting the inspector's sight line on a monitor or the like. The sight line detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing from around the monitor. At this time, the object recognition engine as described above may be used based on pupil detection of the moving image. In addition, the instruction from the examiner may be an instruction through brain waves or weak electrical signals flowing through the body or the like.

[0290] In this case, for example, the training data may be the following training data: character data or voice data (waveform data) indicating instructions for displaying the results obtained by the processing of the above-mentioned various learning models are used as input data, and an execution command for actually making the results obtained by the processing of the various learning models actually displayed on the display unit is used as the correct answer data. In addition, the training data may be the following training data: for example, character data or voice data indicating instructions for displaying high-quality images obtained by the learning model for improving image quality are used as input data, and an execution command for displaying high-quality images and an execution command for changing the button 3420 to an activated state are used as the correct answer data. Of course, any kind of training data may also be used, as long as, for example, the instruction content indicated by the character data or voice data and the execution command content correspond to each other. In addition, the voice data may be converted into character data using an acoustic model or a language model. In addition, the waveform data obtained using a plurality of microphones may be used to reduce the processing of noise data superimposed on the voice data. In addition, the following construction may be adopted: so that, according to the instructions from the examiner, a selection may be made between instructions issued by characters or voices and instructions input using a mouse or a touch panel. In addition, a configuration may be adopted such that selection can be made to turn on or off instructions by characters, voice, or the like according to instructions from the examiner.

[0291] In this case, the machine learning includes deep learning as described above, and, for example, a recurrent neural network (RNN) may be used as at least a part of a multilayer neural network. Here, as an example of a machine learning model according to this variant, reference will be made to 16A and Fig. 16B The RNN as a neural network for processing time series information is described. In addition, reference will be made to 17A and Fig. 17B A long short-term memory (hereinafter referred to as “LSTM”) as a type of RNN is described.

[0292] Fig.16A The structure of RNN as a machine learning model is shown. RNN 3520 has a loop structure in the network, and inputs data x at time t t 3510, and output data h t 3530. Since the RNN 3520 has a loop function in the network, the state of the current time can be taken over to the next state, so the timing information can be processed. Fig. 16B An example of the input / output of a parameter vector at time t is shown. Data x t 3510 includes N data (parameters ms1 to msN). In addition, the data h output by RNN 3520 t3530 includes N pieces of data (parameters ms1 to msN) corresponding to the input data.

[0293] However, since RNN cannot process long-term information during back-propagation, LSTM can be used. LSTM can learn long-term information by providing a forget gate, input gate, and output gate. Fig.17A The structure of LSTM is shown. In LSTM3540, the information that the network takes over at the next time t is the internal state c of the network called the cell. t-1 , and output data h t-1 . Please note that lowercase letters (c, h, x) in the figure represent vectors.

[0294] Next, in Fig. 17B LSTM 3540 is shown in detail in Fig. 17B In FIG. 1 , reference numeral FG denotes a forget gate network, reference numeral IG denotes an input gate network, reference numeral OG denotes an output gate network, and each of these networks is an S-shaped layer. Therefore, a vector whose elements have values ​​from 0 to 1 is output. The forget gate network FG determines how much past information is retained, and the input gate network IG determines which value to update. Reference numeral CU denotes a cell update candidate network, which is an activation function tanh layer. This creates a vector of new candidate values ​​to be added to the cell. The output gate network OG selects the elements of the cell candidate and selects how much information to send at the next time.

[0295] Please note that the above LSTM model is a basic form, and the present invention is not limited to the network shown here. The coupling between the networks can be changed. QRNN (quasi-recurrent neural network) can be used instead of LSTM. In addition, the machine learning model is not limited to neural networks, and Boosting or Support Vector Machine (Support Vector Machine) can be used. In addition, in the case of inputting instructions from the inspector through characters or voice, etc., techniques related to natural language processing (e.g., sequence to sequence) can be applied. In addition, a dialogue engine (dialogue model or learning model for dialogue) that responds to the inspector with outputs such as text or voice can be applied.

[0296] (Variant 10)

[0297] In the above-described various embodiments and modifications, a high-quality image or the like may be stored in a memory according to an instruction from an inspector. At this time, after the inspector instructs to save the high-quality image or the like, when registering a file name, in a state where the file name can be edited according to an instruction from the inspector, the file name may be displayed as a recommended file name, the file name including information (e.g., characters) indicating that the image is an image generated by processing using a learning model for improving image quality (image quality improvement processing) at any part (e.g., the first part or the last part) of the file name.

[0298] Furthermore, when the display unit is caused to display a high-quality image on various display screens such as a report screen, a display indicating that the image being displayed is a high-quality image generated by processing using a learning model for improving image quality may be displayed together with the high-quality image. In this case, since the user can easily discern through the relevant display that the displayed high-quality image is not an actual image obtained by videography, misdiagnosis can be reduced and diagnostic efficiency can be improved. Note that the display indicating that a high-quality image is generated by processing using a learning model for improving image quality may be in any form as long as it is a display that can distinguish between an input image and a high-quality image generated by relevant processing. Furthermore, also for processing using the various learning models described above, not just processing using a learning model for improving image quality, a display indicating that a result being displayed is generated by processing using a relevant kind of learning model may be displayed together with the relevant result.

[0299] At this time, a display screen such as a report screen can be stored in the storage unit in accordance with an instruction from the inspector. For example, the report screen can be stored in the storage unit as a single image in which high-quality images, etc., and a display indicating that these images are generated by processing using a learning model for improving image quality are displayed side by side.

[0300] Furthermore, with respect to a display indicating that a high-quality image is generated by processing using a learning model for improving image quality, a display indicating what kind of training data is used by the learning model for improving image quality when learning is performed may be displayed on the display unit. The display in question may include a description of the types of input data and correct answer data of the training data, or any display related to correct answer data such as imaging sites included in the input data and correct answer data. Note that also with respect to processing using the various learning models described above, not just processing using a learning model for improving image quality, a display indicating what kind of training data is used by the relevant kind of learning model when learning is performed may be displayed on the display unit.

[0301] Furthermore, a configuration may be adopted in which information indicating that an image is generated by processing using a learning model for improving image quality (e.g., characters) can be displayed or stored in a state where the information is superimposed on a high-quality image or the like. At this time, the position where the information is superimposed on the image may be any position as long as the position is in an area (e.g., the edge of the image) that does not overlap with an area where a portion of interest or the like that is a photographing target is displayed. Furthermore, a non-overlapping area may be determined, and the information may be superimposed in the determined area.

[0302] In addition, the following configuration may be adopted: in the case where the default setting is set so that the button 3420 enters the activated state (the image quality improvement process is set to "on") as the initial display screen of the report screen, a report image corresponding to the report screen including the high-quality image, etc. is sent to the server according to an instruction from the examiner. In addition, the following configuration may be adopted: in the case where the default setting is set so that the button 3420 enters the activated state, when the inspection is finished (for example, in the case where the image confirmation screen or the preview screen is changed to the report screen according to an instruction from the examiner), a report image corresponding to the report screen including the high-quality image, etc. is (automatically) sent to the server. At this time, the following configuration may be adopted: a report image generated based on various settings of the default setting (for example, settings related to at least one of the depth range for generating an en-face image on the initial display screen of the report screen, whether to superimpose an analysis diagram, whether the image is a high-quality image, and whether to show a display screen for subsequent observation, etc.) is sent to the server.

[0303] (Variant 11)

[0304] In the various embodiments and modifications described above, in the various learning models described above, an image obtained using a first type of learning model (e.g., a high-quality image, an image showing an analysis result such as an analysis graph, an image showing an object recognition result, or an image showing a segmentation result) may be input to a second type of learning model different from the first type. At this time, the following configuration may be adopted: a result (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result) is generated by processing the second type of learning model.

[0305] In addition, in the various learning models described above, an image to be input into a second type of learning model different from the first type of learning model can be generated from an image input into the first type of learning model by using a result obtained by processing the first type of learning model (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result). At this time, there is a high probability that the generated image is an image suitable as an image for processing the second type of learning model. Therefore, the accuracy of the image obtained when the generated image is input into the second type of learning model (e.g., a high-quality image, an image showing an analysis result such as an analysis graph, an image showing an object recognition result, or an image showing a segmentation result) is enhanced.

[0306] In addition, using the analysis results or diagnosis results obtained by processing the above-mentioned learning model as search keywords, it is possible to retrieve similar images using an external database stored in a server, etc. Note that in the case where multiple images stored in a database have been managed in a state where each feature value of the multiple images has been attached as supplementary information through machine learning, etc., a similar image search engine (similar image search model or learning model for similar image search) using the image itself as a search keyword can be used.

[0307] (Variant 12)

[0308] Note that the processing for generating motion contrast data in the aforementioned embodiment and modification examples is not limited to the configuration in which processing is performed based on the intensity value of a tomographic image. The above-described various processing can be applied to an interference signal obtained with the OCT imaging unit 100, a signal obtained by subjecting an interference signal to Fourier transform, a signal obtained by subjecting a correlation signal to any processing, and tomographic data including a tomographic image based on these signals, etc. In these cases as well, effects similar to those of the aforementioned configuration can be obtained.

[0309] Although an optical fiber system using a coupler as a separation unit is used, a spatial optical system using a collimator and a beam splitter may also be used. In addition, the configuration of the OCT imaging unit 100 is not limited to the above configuration, and some components included in the OCT imaging unit 100 may be configured as components separate from the OCT imaging unit 100.

[0310] In addition, although the interference optical system of the OCT imaging device 100 is configured as a Mach-Zehnder interferometer in the aforementioned embodiment and modification, the interference optical system is not limited thereto. For example, the interference optical system of the OCT device 1 may have a Michelson interferometer.

[0311] In addition, although a spectral domain OCT (SD-OCT) device using an SLD as a light source is described as an OCT device in the above-mentioned embodiments and modifications, the configuration of the OCT device according to the present invention is not limited thereto. For example, the present invention can also be applied to a swept source OCT (SS-OCT) device using a wavelength swept light source capable of sweeping the wavelength of emitted light or any other type of OCT device. Furthermore, the present invention can also be applied to a line-OCT device using line light.

[0312] Furthermore, in the aforementioned embodiment and the modification, the obtaining unit 210 obtains the interference signal obtained by the OCT imaging unit 100 or the three-dimensional tomographic image generated by the image processing unit 220. However, the configuration in which the obtaining unit 210 obtains these signals or images is not limited to the above configuration. For example, the obtaining unit 210 may obtain these signals from a server or an imaging device connected to the control unit via a LAN, a WAN, the Internet, or the like.

[0313] Note that the learning model may be provided in the control unit 200, 900, or 1400 as an image processing device. The learning model may be composed of, for example, a software module executed by a processor such as a CPU. In addition, the learning model may be provided in a separate server connected to the control unit 200, 900, or 1400. In this case, the control unit 200, 900, or 1400 may perform image quality improvement processing using the learning model by connecting to the server including the learning model via any network such as the Internet.

[0314] (Variant 13)

[0315] In addition, the images to be processed by the image processing apparatus or the image processing method according to the various embodiments and modifications described above include medical images obtained using any modality (camera or imaging method). The medical images to be processed may include medical images obtained by any imaging device, etc., as well as images created by the image processing apparatus or the image processing method according to the embodiments and modifications described above.

[0316] In addition, the medical image to be processed is an image of a predetermined part of the subject (examinee), and the image of the predetermined part includes at least a part of the predetermined part of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black and white image or a color image. In addition, the medical image may be an image representing the structure (form) of the predetermined part, or may be an image representing the function of the predetermined part. For example, images representing functions include images representing hemodynamics (blood flow, blood flow velocity, etc.) such as OCTA images, Doppler OCT images, fMRI images, and ultrasound Doppler images. Please note that the predetermined part of the subject can be determined according to the imaging target, and these predetermined parts include organs such as the human eye (eye to be examined), brain, lungs, intestines, heart, pancreas, kidneys, and liver, as well as any parts such as the head, chest, legs, and arms.

[0317] In addition, the medical image may be a tomographic image of the object, or may be a frontal image. Examples of frontal images include a frontal image of the fundus, a frontal image of the anterior segment of the eye, a fundus image obtained by fluorescence imaging, and an en-face image generated using data of at least a partial range of the imaging target in the depth direction for data obtained by OCT (three-dimensional OCT data). The en-face image may be an OCTA en-face image (motion contrast frontal image) generated using data of at least a partial range of the imaging target in the depth direction for three-dimensional OCTA data (three-dimensional motion contrast data). In addition, three-dimensional OCT data or three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0318] In addition, the term "imaging device" refers to a device for performing imaging to obtain an image to be used for diagnosis. Examples of imaging devices include a device that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, and a device that obtains an image of a predetermined portion by detecting radiation emitted from the subject. More specifically, examples of imaging devices according to the various embodiments and modifications described above include at least an X-ray imaging device, a CT device, an MRI device, a PET device, a SPECT device, an SLO device, an OCT device, an OCTA device, a fundus camera, and an endoscope.

[0319] Note that a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device may be included as examples of an OCT device. Furthermore, examples of a Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a swept source OCT (SS-OCT) device. Furthermore, an adaptive optical SLO (AO-SLO) device and an adaptive optical OCT (AO-OCT) device using an adaptive optical system or the like may be included as examples of an SLO device or an OCT device, respectively. Furthermore, a polarization-sensitive SLO (PS-SLO) device and a polarization-sensitive OCT (PS-OCT) device for visualizing information related to a polarization phase difference or depolarization may be included as examples of an SLO device or an OCT device, respectively.

[0320] According to the embodiment and one of the modified examples of the present invention described above, it is possible to generate an image more suitable for image diagnosis than an image generated according to the conventional technique.

[0321] (Other embodiments)

[0322] The present invention can also be implemented by a process in which a program that implements one or more functions according to the above-described embodiments is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. In addition, the present invention can be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0323] The present invention is not limited to the above-described embodiments, and various changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are attached.

[0324] This application claims the benefit of priority from Japanese Patent Application No. 2018-166817 filed on September 6, 2018 and Japanese Patent Application No. 2019-068663 filed on March 29, 2019, which are hereby incorporated by reference herein in their entirety.

[0325] [reference numerals list]

[0326] 200 control unit (image processing device)

[0327] 224 Image Quality Improvement Unit

[0328] 250 Display control unit

Claims

1. An image processing device, comprising: an acquisition unit configured to acquire a plurality of first images of the eye to be inspected at different dates and times; an image quality improving unit configured to generate each of a plurality of second images having at least one of lower noise and higher contrast than each of the plurality of first images obtained, using each of the plurality of first images obtained as input data of a learning model; as well as a display control unit configured to control the display unit to switch between display of the plurality of first images obtained and display of the plurality of second images generated at a plurality of display areas, in which the plurality of display areas of the plurality of first images obtained and the plurality of display areas of the plurality of second images generated correspond to each other, according to a first instruction from an operator, wherein, by changing the depth range from the first depth range to the second depth range according to a second instruction from the operator, the display control unit controls the display unit to change the display from display of a plurality of second images generated corresponding to the first depth range of the eye to be inspected to display of a plurality of second images generated corresponding to the second depth range, the second depth range being at least partially different from the first depth range, The depth range refers to the depth range of the data projected or integrated when generating the second image.

2. The image processing device according to claim 1, further comprising: A selection unit is configured to select a learning model to be used by the image quality improvement unit from among a plurality of learning models based on an imaging condition of the obtained first image.

3. The image processing device according to claim 1, in, The first image is a front image generated based on information about the subject's eye within a range in the depth direction. The image processing device further includes a selection unit configured to select, from a plurality of learning models, a learning model corresponding to a range in a depth direction used for generating the obtained first image, as a learning model to be used by the image quality improvement unit.

4. The image processing apparatus according to claim 3, wherein: When the range in the depth direction used to generate the obtained first image is changed according to an instruction from the operator, the display control unit changes from displaying the first image and the second image being displayed in a juxtaposed manner on the display unit to displaying the first image based on the changed range in the depth direction and the second image generated from the first image on the display unit.

5. The image processing apparatus according to claim 1, wherein: According to an instruction from the operator, the display control unit causes the obtained first image and the generated second image being displayed in a juxtaposed manner on the display unit to be enlarged and displayed together.

6. The image processing apparatus according to claim 1, wherein: The display control unit switches and displays the obtained first image and the generated second image on the display unit according to an instruction from the operator.

7. The image processing apparatus according to claim 6, wherein: The image quality improving unit generates a plurality of second images from the plurality of acquired first images; and According to an instruction from an operator, the display control unit causes the plurality of obtained first images and the plurality of generated second images to be switched and collectively displayed on the display unit.

8. The image processing apparatus according to claim 1, wherein: The display control unit sets transparency for at least one of the obtained first image and the generated second image, and causes the obtained first image and the generated second image to be displayed on the display unit in a superimposed manner.

9. The image processing apparatus according to claim 1, further comprising: a comparing unit configured to compare the obtained first image and the generated second image and generate a colored color map image based on the comparison result, The display control unit causes the color image to be displayed in a superimposed manner on the obtained first image or the generated second image on the display unit.

10. An image processing method, comprising: acquiring a plurality of first images of the eye to be inspected at different dates and times; generating each of the plurality of second images having at least one of lower noise and higher contrast than each of the plurality of first images obtained using each of the plurality of first images obtained as input data for a learning model; as well as controlling the display unit to switch between display of the plurality of first images obtained and display of the plurality of second images generated at a plurality of display areas in accordance with a first instruction from the operator, wherein the plurality of display areas of the plurality of first images obtained and the plurality of display areas of the plurality of second images generated correspond to each other, wherein, by changing the depth range from the first depth range to the second depth range according to a second instruction from the operator, the display unit is controlled to change the display from the display of the plurality of second images generated corresponding to the first depth range of the eye to be inspected to the display of the plurality of second images generated corresponding to the second depth range, the second depth range being at least partially different from the first depth range, The depth range refers to the depth range of the data projected or integrated when generating the second image.

11. A non-transitory computer-readable medium having a program stored thereon, which, when executed by a processor, causes the processor to execute the steps of the image processing method according to claim 10.