Ophthalmic image processing program and ophthalmic image processing device

Abstract

An ophthalmic image processing program and an ophthalmic image processing device are provided that can more appropriately improve the image quality of a motion contrast front image. [Solution] The mathematical model is trained using motion contrast en face image data based on L (L≧2) OCT signals out of multiple OCT signals acquired for the same location on the same tissue as input training data, and motion contrast en face image data based on H (H>L) OCT signals as output training data. A control unit of the ophthalmic image processing device acquires the motion contrast en face image as a base image. The control unit inputs the base image into the mathematical model to acquire a high-quality image output by the mathematical model. The control unit brings each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step closer to each position within the image area of ​​the base image.

Description

[Technical Field]

[0001] The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used to process ophthalmic images of a subject's eye. [Background technology]

[0002] A technique for acquiring various medical information using a mathematical model trained by a machine learning algorithm has been proposed. For example, in an ophthalmologic image processing device described in Patent Document 1, an ophthalmologic image is input as an input image to a mathematical model trained by a machine learning algorithm, and a high-quality image in which the quality of the input image is improved is output by the mathematical model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 045019 Summary of the Invention [Problem to be solved by the invention]

[0004] The inventors of the present invention have conducted extensive research into techniques for improving the image quality of motion contrast en face images. A motion contrast image is an image (e.g., an OCT angiogram) obtained by processing at least two OCT signals acquired by an OCT device at different times for the same location of tissue in a subject's eye, and it displays information about the movement of the subject. A motion contrast en face image is a two-dimensional motion contrast image of tissue viewed from the front along the optical axis of the OCT measurement light. Acquiring a motion contrast en face image requires a longer time than capturing other OCT images (e.g., two-dimensional tomographic images and three-dimensional tomographic images). Therefore, motion contrast en face images are prone to motion artifacts (e.g., image discontinuities and the appearance of unwanted streaks in the image) caused by tissue movement during imaging. Even when tracking is performed to track the imaging position relative to tissue movement, it is difficult to completely eliminate the effects of motion artifacts. Therefore, it is important to suppress the effects of motion artifacts in motion contrast en face images.

[0005] Therefore, the inventors of the present invention considered training a mathematical model using motion contrast enlarged images based on a small number (e.g., two) of multiple OCT signals acquired at different times for the same location on the same tissue as input training data, and motion contrast enlarged images based on a large number of OCT signals as output training data. In this case, while each input training data is likely to contain motion artifacts, the output training data based on the large number of OCT signals is likely to have reduced effects of the motion artifacts. Therefore, by training a mathematical model using images based on the large number of OCT signals as output training data, the effects of motion artifacts in the motion contrast enlarged images are appropriately reduced.

[0006] However, tissue movement often occurs even during capture of motion-contrast en face images based on two OCT signals. Naturally, tissue movement is likely to occur during capture of multiple motion-contrast en face images based on multiple OCT signals. In other words, even between multiple motion-contrast en face images captured at the same position, the positions and shapes of the images (specifically, the objects, such as tissues, depicted in the images) are unlikely to match perfectly. Therefore, the positions and shapes of the images of input training data (input images) and the images of output training data (output images) are also unlikely to match perfectly. As a result, the mathematical model also learns the differences in position and shape between the input and output images, and the learned differences in position and shape may appear in the high-quality images output by the mathematical model. Therefore, a technology capable of more appropriately improving the quality of motion-contrast en face images is desired.

[0007] A typical object of the present disclosure is to provide an ophthalmic image processing program and an ophthalmic image processing device that can more appropriately improve the image quality of a motion contrast front image. [Means for solving the problem]

[0008] An ophthalmic image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of tissues of a subject's eye, and the ophthalmic image processing device is capable of acquiring a high-quality image in which the quality of the input motion contrast en face image is improved by inputting the motion contrast en face image obtained by processing at least two OCT signals acquired by an OCT device at different times for the same position of the tissue of the subject's eye into a mathematical model trained by a machine learning algorithm, and the mathematical model is configured to select L (L≧2) OCT signals from among a plurality of OCT signals acquired at different times for the same position of the same tissue, and The ophthalmologic image processing program is trained using data of a motion contrast front image based on H (H>L) OCT signals as input training data and data of a motion contrast front image based on H (H>L) OCT signals as output training data, and the ophthalmologic image processing device executes the following steps by executing the ophthalmologic image processing program by a control unit of the ophthalmologic image processing device: a base image acquisition step of acquiring a motion contrast front image as a base image; a high-quality image acquisition step of inputting the base image into a mathematical model to acquire a high-quality image output by the mathematical model; and a positioning step of aligning each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step with each position within the image area of ​​the base image.

[0009] An ophthalmic image processing device provided by a typical embodiment of the present disclosure is an ophthalmic image processing device that processes ophthalmic images, which are images of tissues of a subject's eye, and is capable of acquiring a high-quality image by improving the quality of the input motion contrast en face image by inputting a motion contrast en face image obtained by processing at least two OCT signals acquired by an OCT device at different times for the same position of the tissue of the subject's eye into a mathematical model trained by a machine learning algorithm, and the mathematical model is capable of acquiring a high-quality image by improving the quality of the input motion contrast en face image by inputting L (L≧2) OCT signals out of a plurality of OCT signals acquired at different times for the same position of the same tissue. The ophthalmologic image processing device is trained using data of a motion contrast front image based on H (H>L) OCT signals as input training data and data of a motion contrast front image based on H (H>L) OCT signals as output training data, and the control unit of the ophthalmologic image processing device executes a base image acquisition step of acquiring the motion contrast front image as a base image, a high-quality image acquisition step of inputting the base image to a mathematical model to acquire a high-quality image output by the mathematical model, and a positioning step of aligning each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step with each position within the image area of ​​the base image.

[0010] According to the ophthalmological image processing program and ophthalmological image processing device according to the present disclosure, the image quality of a motion contrast front image is more appropriately improved. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the schematic configuration of a mathematical model construction device 1, an ophthalmological image processing device 21, and an OCT device 11. FIG. [Figure 2] FIG. 10 is a diagram showing an example of input training data and output training data when training a mathematical model. [Figure 3] 1 is an enlarged view of five motion contrast front images 600A to 600E taken at the same position and a partial added image 61 of the five motion contrast front images 600A to 600E. [Figure 4] 10 is a flowchart of ophthalmologic image processing executed by the ophthalmologic image processing apparatus 21 of the present embodiment. [Figure 5] 1A to 1C are explanatory diagrams for explaining the process of image processing by the ophthalmologic image processing of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Summary> The ophthalmic image processing device exemplified in the present disclosure can obtain a high-quality image by improving the quality of the input motion contrast front image by inputting a motion contrast front image to a mathematical model trained by a machine learning algorithm. A motion contrast front image is a front image obtained by processing at least two OCT signals acquired by an OCT device at different times for the same location on the tissue of the subject's eye. The mathematical model is trained using motion contrast front image data based on L (L≧2) OCT signals among multiple OCT signals acquired at different times for the same location on the same tissue as input training data, and motion contrast front image data based on H (H>L) OCT signals as output training data. The control unit of the ophthalmic image processing device executes a base image acquisition step, a high-quality image acquisition step, and a registration step. In the base image acquisition step, the control unit acquires the motion contrast front image as a base image (an image before being enhanced by the mathematical model). In the high-quality image acquisition step, the control unit inputs the base image to the mathematical model to obtain a high-quality image output by the mathematical model. In the positioning step, the control unit brings each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step closer to each position within the image area of ​​the original image.

[0013] As described above, when training a mathematical model, images based on a large number of OCT signals are used as output training data, allowing the mathematical model to output a high-quality image in which the effects of motion artifacts are appropriately reduced. Meanwhile, the mathematical model also learns the position and shape differences between the input training data and the output training data. As a result, the position and shape of the high-quality image output by the mathematical model are misaligned with the position and shape of the base image input to the mathematical model. Therefore, when comparing the base image and the high-quality image directly, the user may feel uncomfortable or uneasy. In contrast, the ophthalmic image processing device disclosed herein aligns each position within the image area of ​​the high-quality image output by the mathematical model with each position within the image area of ​​the base image. This reduces the user's discomfort when comparing the base image and the high-quality image. In other words, the technology disclosed herein appropriately suppresses the influence of learning the position and shape differences between the input training data and the output training data. This facilitates more appropriate improvement in the image quality of motion contrast frontal images.

[0014] In this disclosure, the expression "aligning" each position within the image area of ​​the high-quality image with each position within the image area of ​​the base image is sometimes used. However, this expression is not limited to strictly aligning the positions, but also includes bringing the positions closer together.

[0015] This disclosure illustrates a case where the OCT device itself functions as an ophthalmic image processing device that processes ophthalmic images. In this case, the OCT device can process the captured ophthalmic images while capturing them. For example, the OCT device can acquire a high-quality image of the captured ophthalmic image immediately after capturing the image and allow the user to review it. However, devices that can function as ophthalmic image processing devices are not limited to OCT devices. For example, a personal computer (hereinafter referred to as a "PC") may be used as the ophthalmic image processing device. In this case, the PC may acquire data of the ophthalmic images captured by the OCT device via at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), etc., and perform processing on the acquired ophthalmic images. Furthermore, control units of multiple devices may cooperate to process ophthalmic images.

[0016] A specific method for aligning each position in the image area of ​​the high-quality image with each position in the image area of ​​the original image (i.e., aligning the two images) can be selected as appropriate. For example, the control unit may search for a corresponding local area in the other image for each of multiple local areas set in one of the original image and the high-quality image using various methods (e.g., phase-only correlation, etc.). The control unit may calculate the movement direction and movement amount of each local area based on the search result. At this time, interpolation and smoothing using a thin plate spline method, etc. may also be performed.

[0017] Here, the control unit may perform the alignment process for each local region including multiple pixels, rather than for each pixel. If the alignment process is performed for each pixel, there is a possibility that motion artifacts contained in the original image before the image quality improvement using the mathematical model will be restored by the alignment process. In contrast, performing the alignment process for each local region including multiple pixels reduces the possibility of motion artifacts being restored.

[0018] However, the registration method can be changed. For example, the high-quality image may be registered with the base image by using techniques such as LDDMM (Large Deformation Diffeomorphic Metric Mpping), Optical Flow, or Patch Match.

[0019] In this disclosure, an example is given of a case where alignment is performed by moving each position within the image area of ​​the high-quality image so that it approaches each position within the image area of ​​the base image. In this case, each position of the high-quality image output by the mathematical model is moved closer to each position of the actually captured base image, so the high-quality image is likely to be an image that is closer to the actual tissue condition. Furthermore, for example, even if an actually captured base image is first displayed on a display unit and then a high-quality image aligned with the base image is displayed, the positions of the previously displayed base image do not change (become distorted) during display. However, for example, it is also possible to move each position within the image area of ​​the base image so that it approaches each position within the image area of ​​the high-quality image. Even in this case, the user is less likely to feel uncomfortable when comparing the base image and the high-quality image.

[0020] The control unit may further perform a brightness correction step of approximating the brightness at each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step to the brightness at each corresponding position within the image area of ​​the original image.

[0021] In this case, not only are each position within the image area of ​​the high-quality image aligned with each position within the image area of ​​the base image, but the brightness of each position between the two images is also approximated. Therefore, the user is less likely to feel uncomfortable when comparing the base image and the high-quality image. Furthermore, even if the mathematical model erroneously generates a vascular structure or the like from noise contained in the base image, the brightness of each position between the high-quality image and the base image is approximated, thereby appropriately preventing the two images from appearing extremely different.

[0022] A specific method for approximating the luminance at each position of the high-quality image and the original image can also be selected as appropriate. For example, the control unit may calculate the average luminance within a local region at the same position for each of the high-quality image and the original image after the above-mentioned alignment is completed, and adjust the luminance within at least one of the local regions so that the average luminances approach (for example, match) each other.

[0023] In this disclosure, an example is given in which the luminance of each position in the high-quality image is made closer to the luminance of each position in the base image. In this case, the luminance of each position in the high-quality image output by the mathematical model is made closer to the luminance of each position in the actually captured base image, so the high-quality image is more likely to resemble the actual tissue condition. Furthermore, for example, even if an actually captured base image is first displayed on a display unit and then a high-quality image with a luminance adjusted to that of the base image is displayed, the luminance of each position in the previously displayed base image does not change during display. However, for example, it is also possible to make the luminance of each position in the base image closer to the luminance of each position in the high-quality image. Even in this case, the user is less likely to feel uncomfortable when comparing the base image and the high-quality image. Furthermore, the effect of non-existent tissues being generated based on noise is appropriately suppressed.

[0024] The mathematical model may be pre-trained to receive a base image and output the high-quality image in which motion artifacts contained in the base image have been reduced. Because capturing a motion-contrast front image takes a long time, motion artifacts are likely to be present in the motion-contrast front image. However, by training the mathematical model using images based on a large number of OCT signals as output training data, the mathematical model can appropriately reduce motion artifacts contained in the base image. Furthermore, by aligning the high-quality image in which motion artifacts have been reduced with the actually captured base image, the influence of learning differences in position and shape between the input training data and the output training data is appropriately suppressed. This facilitates more appropriate improvement of the image quality of the motion-contrast front image.

[0025] The motion artifacts in the original image that are reduced by the mathematical model may be, for example, image discontinuities (sometimes referred to as image jerks) and unwanted streaks in the image. The mathematical model may output a high-quality image in which noise contained in the original image has also been reduced. The mathematical model may also output a high-quality image in which the contrast of the original image has also been improved.

[0026] <Embodiment> (Device configuration) A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, this embodiment uses a mathematical model construction device 1, an ophthalmic image processing device 21, and an OCT device 11. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. A program for realizing the constructed mathematical model is stored in a storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 inputs an ophthalmic image as a base image (input image) to the mathematical model, thereby obtaining a high-quality image by improving the quality of the base image. The ophthalmic image processing device 21 also performs registration between the base image and the high-quality image. The OCT device 11 captures ophthalmic images, which are images of the tissues of the subject's eye.

[0027] As an example, a personal computer (hereinafter referred to as a "PC") is used as the mathematical model construction device 1 of this embodiment. As will be described in detail later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using ophthalmic images acquired from the OCT device 11 (hereinafter referred to as "training ophthalmic images") and images obtained by improving the image quality of the training ophthalmic images (e.g., arithmetic average images, etc.). However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the OCT device 11 may function as the mathematical model construction device 1. Furthermore, control units of multiple devices (e.g., the CPU of the PC and the CPU 13 of the OCT device 11) may cooperate to construct a mathematical model.

[0028] Furthermore, the ophthalmic image processing device 21 of this embodiment has a configuration for capturing ophthalmic images similar to images captured by the OCT device 11. That is, in this embodiment, the OCT device itself functions as the ophthalmic image processing device 21 that processes ophthalmic images. Therefore, the ophthalmic image processing device 21 of this embodiment can process the captured ophthalmic images while capturing them. However, devices that can function as ophthalmic image processing devices are not limited to OCT devices. For example, a PC may be used as the ophthalmic image processing device. Furthermore, control units of multiple devices may work together to process ophthalmic images.

[0029] In addition, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than a CPU may be used for at least some of the various devices. For example, a GPU may be used as a controller to speed up processing.

[0030] The mathematical model construction device 1 will now be described. The mathematical model construction device 1 is installed, for example, at a manufacturer that provides an ophthalmic image processing device 21 or an ophthalmic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller responsible for control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing the mathematical model construction process described below. The communication I / F 5 also connects the mathematical model construction device 1 to other devices (for example, an OCT device 11 and an ophthalmic image processing device 21, etc.).

[0031] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the mathematical model construction device 1. The operation unit 7 can be, for example, at least one of a keyboard, a mouse, a touch panel, etc. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 7. The display device 8 displays various images. The display device 8 can be any of a variety of devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.). Note that, in this disclosure, "image" includes both still images and moving images.

[0032] The mathematical model construction device 1 can acquire data of ophthalmic images (motion contrast front images in this embodiment) from the OCT device 11. The mathematical model construction device 1 may acquire data of ophthalmic images from the OCT device 11 by, for example, at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), and the like.

[0033] The ophthalmic image processing device (OCT device) 21 will be described. The ophthalmic image processing device 21 is installed, for example, in a facility (e.g., a hospital or a health checkup facility) where a diagnosis or examination of a subject is performed. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23, which is a controller responsible for control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing the ophthalmic image processing described below. The ophthalmic image processing program includes a program for realizing a mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (e.g., the mathematical model construction device 1, etc.). Furthermore, the ophthalmic image processing device 21 of this embodiment includes an OCT unit 26 that captures ophthalmic images (motion contrast frontal images in this embodiment) similar to the images captured by the OCT device 11. The OCT section 26 can have the same configuration as the OCT section 16 (details of which will be described later) included in the OCT device 11.

[0034] The ophthalmologic image processing device 21 is connected to an operation unit 27 and a display device 28. As with the operation unit 7 and display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0035] The OCT device 11 will be described. As an example, in this embodiment, a case will be described in which the OCT device 11 that provides ophthalmic images to the mathematical model construction device 1 and an OCT device that functions as an ophthalmic image processing device 21 are used. However, the number of OCT devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from multiple OCT devices. Alternatively, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a single shared OCT device.

[0036] The OCT device 11 includes a control unit 12 that performs various control processes, and an OCT section 16. The control unit 12 includes a CPU 13 that is a controller that manages control, and a storage device 14 that can store programs, data, and the like. The OCT section 16 includes various components necessary for capturing ophthalmologic images of the subject's eye. The OCT section 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning section that scans the measurement light, an optical system that irradiates the subject's eye with the measurement light, and a light-receiving element that receives a composite light of the light reflected by the tissue of the subject's eye and the reference light.

[0037] The OCT device 11 and the ophthalmologic image processing device 21 can capture two-dimensional tomographic images and three-dimensional tomographic images of tissue of the subject's eye (for example, fundus tissue in this embodiment). In particular, the CPU 13 scans the OCT light (measurement light) along a scan line to capture two-dimensional tomographic images of a cross section intersecting the scan line. The CPU 13 can also capture three-dimensional tomographic images of the tissue by two-dimensionally scanning the OCT light.

[0038] Furthermore, the OCT device 11 and the ophthalmologic image processing device 21 can capture a motion contrast en face image of the tissue of the subject's eye (for example, the fundus tissue in this embodiment). A motion contrast image is an image acquired by processing multiple OCT signals acquired at different times for the same position. The motion contrast image contains information about the movement of the subject. Therefore, the motion contrast image of this embodiment allows the blood vessels in the fundus of the subject's eye to be grasped non-invasively. In this embodiment, a two-dimensional motion contrast en face image is captured and processed, in which the biological tissue is viewed from the front direction along the optical axis of the OCT measurement light. The two-dimensional motion contrast en face image may be a two-dimensional en face image in which the captured area of ​​the three-dimensional image is viewed from the front direction. The data of the Enface image may be, for example, integrated image data in which brightness values ​​are integrated in the depth direction (Z direction) at each position in the direction (XY direction) intersecting the optical axis of the OCT measurement light, integrated values ​​of spectral data at each position in the XY direction, brightness data at each position in the XY direction in a certain depth direction, brightness data at each position in the XY direction in any layer of biological tissue (e.g., the retinal surface layer, etc.), etc. The OCT device 11 and the ophthalmologic image processing device 21 can also perform tracking processing to make the scanning position of the OCT light follow the movement of the subject's eye.

[0039] (Mathematical model construction process) 2, a description will be given of a mathematical model construction process executed by the mathematical model construction device 1. The mathematical model construction process is executed by the CPU 3 in accordance with a mathematical model construction program stored in the storage device 4.

[0040] In the mathematical model construction process, a mathematical model is trained using a training dataset to construct a mathematical model that outputs a high-quality image obtained by improving (converting) the quality of a base image (input image). The training dataset includes input data (input training data) and output data (output training data). The mathematical model is capable of converting various ophthalmic images into high-quality images. The type of training dataset used to train the mathematical model is determined depending on the type of ophthalmic image whose quality is to be converted by the mathematical model. In this embodiment, a two-dimensional motion-contrast front image of the fundus taken from the front is input to the mathematical model as a base image, and a motion-contrast front image (high-quality image) obtained by improving the quality of the base image is output to the mathematical model as a converted image.

[0041] Specifically, the input training data in this embodiment uses data of motion contrast en face images based on a small number, L (L≧2), of OCT signals acquired at different times for the same position of the same tissue, and the output training data uses data of motion contrast en face images based on a large number, H (H>L), of the aforementioned OCT signals.

[0042] The method for generating the output training data can be selected as appropriate. For example, the output training data may be obtained by performing an addition process (which may be an averaging process) on three-dimensional volume data generated based on each of H OCT signals and generating a motion contrast front image from the three-dimensional volume data obtained by the addition process. Alternatively, a plurality of motion contrast front images may be generated based on L OCT signals from among the plurality of OCT signals. As described above, at least one of the plurality of motion contrast front images based on L OCT signals may be used as input training data. The plurality of motion contrast front images based on L OCT signals may be added (which may be an averaging process) to generate data on a motion contrast front image based on H (H>L) OCT signals as output training data. The following description will be given taking as an example a case where an added image of a plurality of motion contrast front images is used as output training data.

[0043] FIG. 2 shows an example of input training data and output training data when outputting a high-quality image to a mathematical model. In the example shown in FIG. 2, the CPU 3 acquires a set 60 of a plurality of motion contrast frontal images 600A to 600X that are images of the same position of the same tissue. In the example shown in FIG. 2, each of the motion contrast frontal images 600A to 600X is generated based on two OCT signals acquired for the same position. The CPU 3 uses a part of the plurality of motion contrast frontal images 600A to 600X in the set 60 (a number “f” of motion contrast frontal images that is less than the number “m” used for the additive average of the output training data described later) as the input training data. In the present embodiment, one motion contrast frontal image 600A is used as the input training data. However, an additive image of f (f < m) motion contrast frontal images may be used as the input training data. Further, the CPU 3 acquires an additive image 61 (additive average image in the present embodiment) of m (m > f) motion contrast frontal images 600A to 600X in the set 60 as the output training data. That is, in the present embodiment, the number L of OCT signals used for the input training data is “f × 2”, and the number H of OCT signals used for the output training data is “m × 2”, so the relationship “H > L” holds. When the mathematical model is trained with the input training data and the output training data illustrated in FIG. 2, when a motion contrast frontal image is input to the trained mathematical model as a base image, a high-quality motion contrast frontal image is output by the mathematical model.

[0044] The mathematical model construction process will be described. The CPU 3 acquires a part (a number f of motion contrast frontal images) of the plurality of motion contrast frontal images 600A to 600X photographed by the OCT device 11 as the input training data. Next, the CPU 3 acquires an additive image 61 of m (m > f) motion contrast frontal images 600A to 600X.

[0045] Next, the CPU 3 trains a mathematical model using the training dataset using a machine learning algorithm, such as a neural network, a random forest, boosting, or a support vector machine (SVM).

[0046] Neural networks are a method of imitating the behavior of biological neuronal networks. Examples of neural networks include feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0047] Random forest is a method to generate a large number of decision trees by learning from randomly sampled training data. When using random forest, the branches of multiple decision trees that have been trained as classifiers are traced, and the results obtained from each decision tree are averaged (or voted by majority vote).

[0048] Boosting is a technique for generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple weak classifiers.

[0049] SVM is a method for constructing a two-class pattern classifier using linear input elements. SVM learns the parameters of the linear input elements based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

[0050] A mathematical model refers to, for example, a data structure for predicting the relationship between input training data and output training data. A mathematical model is constructed by training using a training dataset. As described above, a training dataset is a set of input training data and output training data. For example, the correlation data (e.g., weights) between each input and output is updated through training.

[0051] In this embodiment, a multi-layer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also referred to as units) are arranged in each layer. In detail, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used. However, other machine learning algorithms may also be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be adopted as the machine learning algorithm.

[0052] The above process is repeated until the construction of the mathematical model is completed. When the construction of the mathematical model is completed, a program and data for realizing the constructed mathematical model are installed in the ophthalmologic image processing device 21.

[0053] (Characteristics of the constructed mathematical model) The characteristics of the mathematical model of this embodiment will be described with reference to FIG. 3. FIG. 3 shows five motion-contrast anterior images 600A-600E captured at the same position and a partially enlarged view of an additive image 61 of the five motion-contrast anterior images 600A-600E. It takes longer to capture each of the motion-contrast anterior images 600A-600E than to capture other OCT images (e.g., two-dimensional tomographic images and three-dimensional tomographic images). Therefore, as shown in FIG. 3, each of the motion-contrast anterior images 600A-600E is prone to motion artifacts (e.g., image discontinuities and unwanted streaks in the image) due to tissue movement during imaging. In fact, each of the motion-contrast anterior images 600A-600E shown in FIG. 3 contains horizontal image shifts and horizontal white streaks. However, motion artifacts are reduced in the additive image 61 (an additive average image in this embodiment) of the five motion-contrast anterior images 600A-600E. Therefore, by training the mathematical model using images based on a large number (L in this embodiment) of OCT signals acquired at the same position as output training data, a high-quality image with appropriately reduced motion artifacts can be obtained. In other words, the mathematical model can output a high-quality image in which motion artifacts contained in an input motion contrast front image (base image) have been appropriately reduced. Furthermore, the mathematical model of this embodiment reduces noise contained in the input base image and outputs a high-quality image in which the contrast of the base image has been improved.

[0054] However, as shown in FIG. 3 , even among multiple motion-contrast front images 600A-600E captured at the same position, the positions and shapes of the images are unlikely to match perfectly. Furthermore, the positions and shapes of the motion-contrast front images 600A-600E and the additive image 61 are also unlikely to match perfectly. As a result, the mathematical model of this embodiment learns the position and shape differences between the input training data image (input image) and the output training data additive image (output image). The learned position and shape differences may appear in the high-quality image output by the mathematical model. For example, the position and shape of the high-quality image output by the mathematical model may differ from the position and shape of the base image input to the mathematical model. Furthermore, the mathematical model may erroneously generate tissues (e.g., vascular structures) in the high-quality image that were not actually present in the image due to noise, etc., contained in the base image. The ophthalmologic image processing device 21 of this embodiment solves the above problems and more appropriately improves the quality of motion-contrast front images.

[0055] (Ophthalmological Image Processing) 4 and 5, an example of ophthalmic image processing executed by the ophthalmic image processing device 21 will be described. The ophthalmic image processing illustrated in Fig. 4 is executed by the CPU 23 in accordance with an ophthalmic image processing program stored in the storage device 24.

[0056] 4, the CPU 23 acquires a motion contrast front image of the tissue of the subject's eye captured by an OCT device (in this embodiment, the motion contrast front image is based on two OCT signals acquired at the same position) as a base image (S1). As described above, in this embodiment, the ophthalmologic image processing device 21 also functions as the OCT device. Therefore, in S1, the CPU 23 controls the OCT unit 26 to capture the motion contrast front image and acquires data of the captured motion contrast front image as data of the base image.

[0057] The CPU 23 inputs the original image acquired in S1 into the mathematical model to acquire a high-quality image output by the mathematical model (S2). As shown in Figure 5, the high-quality image has reduced effects of motion artifacts, reduced noise, and improved contrast compared to the original image.

[0058] Next, the CPU 23 performs a registration process between the base image and the high-resolution image (S3). That is, in S3, the CPU 23 aligns each position within the image region of the high-resolution image with each position within the image region of the base image. As described above, the position and shape of the high-resolution image output by the mathematical model are misaligned with the position and shape of the base image input to the mathematical model. Therefore, when a user directly compares the base image and the high-resolution image, the user may feel uncomfortable or uneasy. In contrast, in this embodiment, a registration process between the base image and the high-resolution image (S3) is performed. Therefore, the user is less likely to feel uncomfortable or uneasy when comparing the base image and the high-resolution image in an aligned state. In other words, the technology disclosed herein appropriately suppresses the influence of learning differences in position and shape between the input training data and the output training data. This facilitates more appropriate improvement in the image quality of motion contrast front images.

[0059] As an example, in S3 of this embodiment, the CPU 23 searches for a corresponding local area in the image area of ​​the high-quality image for each of multiple local areas set in the image area of ​​the base image using various methods (phase-only correlation in this embodiment). Based on the search results, the CPU 23 calculates the movement direction and amount of each local area in the high-quality image. During this process, interpolation and smoothing using a thin plate spline method or the like are also performed. Here, in this embodiment, the CPU 23 performs the alignment process for each local area including multiple pixels, rather than for each pixel. If the alignment process were performed for each pixel, motion artifacts contained in the base image before image enhancement using a mathematical model may be reappeared by the alignment process. In contrast, performing the alignment process for each local area including multiple pixels reduces the possibility of motion artifacts being reappeared. Furthermore, in this embodiment, the alignment is performed by moving each position in the image area of ​​the high-quality image so that it approaches each position in the image area of ​​the base image. In this case, the positions of the high-quality image output by the mathematical model are brought closer to the positions of the actually captured base image, so the high-quality image tends to be closer to the actual state of the tissue.

[0060] Next, the CPU 23 performs a luminance correction process between the base image and the high-quality image to obtain a final image (see FIG. 5) with the image quality enhancement completed (S4). That is, in S4, the CPU 23 approximates the luminance at each position within the image area of ​​the high-quality image to the luminance at each corresponding position within the image area of ​​the base image. As a result, the user is less likely to feel uncomfortable when comparing the base image and the high-quality image after luminance correction. Furthermore, even if the mathematical model erroneously generates tissue (e.g., vascular structure) that was not actually captured in the image due to noise or the like contained in the base image, approximating the luminance at each position between the high-quality image and the base image appropriately prevents the two images from appearing extremely different.

[0061] As an example, in S4 of this embodiment, the CPU 23 calculates the average brightness within a local region at the same position for each of the high-quality image and the base image after the alignment in S3 has been completed, and adjusts the brightness within at least one of the local regions so that the average brightnesses approach (e.g., match). In this embodiment, the brightness within the local region of the high-quality image is adjusted while maintaining the brightness of the base image. In other words, the brightness of each position of the high-quality image output by the mathematical model is made to approach the brightness of each position of the actually captured base image. Therefore, the high-quality image is likely to be an image that closely resembles the actual tissue condition.

[0062] The techniques disclosed in the above embodiments are merely examples. Therefore, it is possible to modify the techniques exemplified in the above embodiments. First, it is possible to execute only some of the techniques exemplified in the above embodiments. For example, in the ophthalmologic image processing shown in FIG. 4, it is possible to omit the luminance correction process (S4) and execute the alignment process (S3). Conversely, it is possible to omit the alignment process (S3) and execute the luminance correction process (S4). [Explanation of symbols]

[0063] 11 OCT device 21 Ophthalmological image processing device 23 CPU 24 Storage device 61 Additive Image 600A~600X Motion Contrast Front Image

Claims

1. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image, which is an image of tissue of a subject's eye, The ophthalmologic image processing device includes: By inputting a motion contrast front image obtained by processing at least two OCT signals acquired by an OCT device at different times for the same position of tissue in the test eye into a mathematical model trained by a machine learning algorithm, it is possible to obtain a high-quality image in which the image quality of the input motion contrast front image is improved; The mathematical model is Among a plurality of OCT signals acquired at different times for the same position of the same tissue, data of a motion contrast front image based on L (L≧2) OCT signals is used as input training data, and data of a motion contrast front image based on H (H>L) OCT signals is used as output training data, The ophthalmologic image processing program is executed by a control unit of the ophthalmologic image processing device, a base image acquisition step of acquiring a motion contrast front image as a base image; a high-quality image acquisition step of inputting the base image into a mathematical model to acquire a high-quality image output by the mathematical model; a positioning step of bringing each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step closer to each position within the image area of ​​the original image; 10. An ophthalmologic image processing program, wherein the program is executed by the ophthalmologic image processing device.

2. 2. The ophthalmologic image processing program according to claim 1, a brightness correction step of approximating the brightness at each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step to the brightness at each corresponding position within the image area of ​​the original image; 10. An ophthalmologic image processing program, wherein the program is executed by the ophthalmologic image processing device.

3. 3. The ophthalmologic image processing program according to claim 1, An ophthalmological image processing program characterized in that the mathematical model is pre-trained to output the high-quality image in which motion artifacts contained in the base image are reduced when the base image is input.

4. An ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of a subject's eye, By inputting a motion contrast front image obtained by processing at least two OCT signals acquired by an OCT device at different times for the same position of tissue in the test eye into a mathematical model trained by a machine learning algorithm, it is possible to obtain a high-quality image in which the image quality of the input motion contrast front image is improved; The mathematical model is Among a plurality of OCT signals acquired at different times for the same position of the same tissue, data of a motion contrast front image based on L (L≧2) OCT signals is used as input training data, and data of a motion contrast front image based on H (H>L) OCT signals is used as output training data, The control unit of the ophthalmologic image processing device a base image acquisition step of acquiring a motion contrast front image as a base image; a high-quality image acquisition step of inputting the base image into a mathematical model to acquire a high-quality image output by the mathematical model; a positioning step of bringing each position within the image area of ​​the high-quality image acquired in the high-quality image acquisition step closer to each position within the image area of ​​the original image; An ophthalmological image processing device characterized by executing the above.

Images