Ophthalmic image processing program and ophthalmic image processing device

Abstract

A control unit of this ophthalmic image processing device executes an image acquisition step (S11), a transformed image acquisition step (S12), and an evaluation information acquisition step (S13). In the image acquisition step, the control unit acquires an ophthalmic image captured by an ophthalmic image imaging device. In the transformed image acquisition step, the control unit enters, as an input image, the ophthalmic image acquired in the image acquisition step to a mathematical model that has been trained by a machine learning algorithm, so as to acquire a transformed image that is obtained by transforming the image quality of the input image. In the evaluation information acquisition step, the control unit acquires evaluation information for evaluating the validity of transformation from the input image into the transformed image by using the mathematical model.

Description

Technical Field

[0001] The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used for processing ophthalmic images of an eye to be examined.

Background Art

[0002] Techniques for obtaining various medical information using a mathematical model trained by a machine learning algorithm have been proposed. For example, in the ophthalmic apparatus described in Patent Document 1, by inputting an eye shape parameter into a mathematical model trained by a machine learning algorithm, IOL-related information (for example, predicted postoperative anterior chamber depth) of the eye to be examined is obtained. Based on the obtained IOL-related information, the IOL power is calculated.

[0003] Also, in Non-Patent Document 1, by inputting an ophthalmic image as an input image into a mathematical model trained by a machine learning algorithm, a converted image obtained by converting the image quality of the input image is obtained.

Prior Art Documents

Non-Patent Documents

[0004]

Patent Document 1

Non-Patent Document 1

Summary of the Invention

[0005] The conversion from input images to transformed images by mathematical models may not always be performed correctly. For example, if the ophthalmic images used to train the mathematical model differ significantly from the ophthalmic images actually input into the mathematical model, the conversion from input images to transformed images may not be performed correctly. If the transformed image is presented to the user without proper conversion, the user may not be able to accurately make various decisions based on the transformed image.

[0006] A typical objective of this disclosure is to provide an ophthalmic image processing program and an ophthalmic image processing device that can present more appropriate information to the user.

[0007] An ophthalmic image processing program provided in a typical embodiment of this disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of tissue of an eye being examined, wherein the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device, and comprises: an image acquisition step of acquiring an ophthalmic image taken by an ophthalmic image acquisition device; a converted image acquisition step of acquiring a converted image in which the effect of speckle noise in the input image is suppressed and the image quality is converted by inputting the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm; and difference information of pixel values ​​between corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model. Take To benefit difference An information acquisition step, and a difference image display step in which a difference image, which is an image of the distribution of the difference information, is displayed on the display unit, The steps include: displaying information on the display unit indicating the degree of validity of the transformation from the input image to the transformed image by the mathematical model, based on the similarity between the difference image and the input image; This is performed by the ophthalmic image processing device.

[0008] An ophthalmic image processing device provided in a typical embodiment of this disclosure is an ophthalmic image processing device that processes ophthalmic images, which are images of tissue of an eye being examined, wherein the control unit of the ophthalmic image processing device includes: an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capture device; a converted image acquisition step of acquiring a converted image by inputting the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm, thereby suppressing the effect of speckle noise in the input image and converting the image quality; and difference information of pixel values ​​between corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model. Take To benefit difference An information acquisition step, and a difference image display step in which a difference image, which is an image of the distribution of the difference information, is displayed on the display unit, The steps include: displaying information on the display unit indicating the degree of validity of the transformation from the input image to the transformed image by the mathematical model, based on the similarity between the difference image and the input image; Execute this.

[0009] According to the ophthalmic image processing program and ophthalmic image processing device described herein, more appropriate information is presented to the user.

[0010] The control unit of the ophthalmic image processing apparatus illustrated in this disclosure performs an image acquisition step, a converted image acquisition step, and an evaluation information acquisition step. In the image acquisition step, the control unit acquires an ophthalmic image captured by an ophthalmic imaging device. In the converted image acquisition step, the control unit inputs the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm, thereby acquiring a converted image with a transformed image quality. In the evaluation information acquisition step, the control unit acquires evaluation information to evaluate the validity of the transformation from the input image to the converted image by the mathematical model.

[0011] According to the ophthalmic image processing device exemplified in this disclosure, evaluation information is obtained to assess the validity of the mathematical model-based transformation from an input image to a transformed image. Therefore, the ophthalmic image processing device can use the evaluation information to present appropriate information to the user.

[0012] Various ophthalmic images can be used as input images. For example, at least one of the following may be used as input images: tomographic images (two-dimensional or three-dimensional tomographic images) taken by an OCT device, images taken by a fundus camera, images taken by a laser scanning optoscope (SLO), and images taken by a corneal endothelial cell imaging device. The ophthalmic image may also be an OCT angio image of the fundus of the eye being examined, taken by an OCT device. The OCT angio image may be a two-dimensional frontal image of the fundus viewed from the front (i.e., in the direction of the line of sight of the eye being examined). The OCT angio image may also be a motion contrast image obtained by processing at least two OCT signals acquired at different times for the same position. The ophthalmic image may also be an Enface image (OCT frontal image) of at least a portion of a three-dimensional tomographic image taken by an OCT device, viewed from a direction along the optical axis of the measurement light of the OCT device (frontal direction).

[0013] Furthermore, the image quality transformed by the mathematical model can be selected as appropriate. For example, the control unit may use the mathematical model to obtain a transformed image in which at least one of the following has been transformed: noise level, contrast, and resolution of the input image.

[0014] In the evaluation information acquisition step, the control unit may acquire, as evaluation information, the difference information between corresponding pixel values ​​in the input image input to the mathematical model and the converted image output from the mathematical model. If the image quality of the input image is converted appropriately, the difference between the input image and the converted image will be small. Therefore, by acquiring the difference information as evaluation information, the validity of the conversion from the input image to the converted image can be appropriately evaluated. The difference information may be the difference in pixel values ​​between corresponding pixels, or it may be the ratio of one pixel value to the other pixel value.

[0015] Furthermore, the control unit may, in the evaluation information acquisition step, perform a Fourier transform (e.g., a two-dimensional Fourier transform) on both the input image and the transformed image, and then acquire the difference information. Periodic artifacts may occur in the transformed image. When periodic artifacts occur in the transformed image, the input image and the transformed image exhibit different frequency distributions. Therefore, by using the Fourier transform, the presence or absence of periodic artifacts can be evaluated more appropriately.

[0016] The control unit may perform a first evaluation step in which, in the difference image obtained by imaging the difference information, if the difference value within an arbitrary region containing multiple pixels is greater than or equal to a threshold, the conversion from the input image to the converted image is evaluated as not valid. Even when the conversion is performed appropriately, the difference image may contain scattered pixels with large values. On the other hand, if the conversion is not performed appropriately due to the presence of diseased areas, etc., regions with a high concentration of pixels with large values ​​will appear in the difference image. Therefore, by comparing the difference value within an arbitrary region containing multiple pixels in the difference image with a threshold, the control unit can suppress the influence of scattered pixels even when the conversion is performed appropriately and appropriately evaluate the validity of the conversion.

[0017] The control unit may perform a smoothing process on the pixel values ​​of the difference image (the difference value corresponding to each pixel) and then evaluate the validity of the conversion based on the difference image. In this case, even if the conversion is performed appropriately, the influence of scattered pixels will be suppressed more effectively.

[0018] Furthermore, specific methods for evaluating the validity of the conversion based on the difference values ​​within the region can be appropriately selected. For example, the control unit may evaluate the conversion as invalid if the average value of the difference values ​​within the region is greater than or equal to a threshold. Alternatively, the control unit may evaluate whether the conversion is valid based on the number of pixels within a unit region whose difference values ​​are greater than or equal to a threshold. The control unit may also acquire information indicating the degree of validity of the conversion based on the difference information. This information indicating the degree of validity may be displayed on the display unit.

[0019] In the evaluation information acquisition step, the control unit may acquire a difference image, which is an image of the difference in pixel values ​​between corresponding pixels in the input image input to the mathematical model and the transformed image output from the mathematical model, and may also acquire the similarity (e.g., correlation) between the difference image and the input image as evaluation information. If the image quality of the input image is appropriately converted, the difference between the input image and the transformed image will be small, and therefore the similarity between the difference image and the input image will be small. On the other hand, if the conversion of the input image is not performed appropriately and irregular parts in the input image affect the conversion, the location of the irregular parts in the input image will be similar to the location in the difference image where the difference value is large, and therefore the similarity between the input image and the difference image will be large. Thus, by acquiring the similarity between the difference image and the input image as evaluation information, the validity of the conversion can be appropriately evaluated. The method of acquiring the similarity can be selected as appropriate. For example, a correlation diagram may be acquired, or a correlation coefficient may be acquired.

[0020] The control unit may further perform a second evaluation step in which it evaluates that the conversion from the input image to the converted image is not valid if the similarity value is above a threshold. As mentioned above, when the conversion is performed properly, the similarity between the input image and the difference image decreases. On the other hand, when the conversion is not performed properly, the similarity between the input image and the difference image increases. Therefore, the control unit can appropriately evaluate whether the conversion from the input image to the converted image is valid by determining whether the similarity value is above a threshold. Various values ​​(e.g., correlation coefficient) can be used as the similarity value as appropriate.

[0021] The control unit may further perform a difference image display step, which involves displaying the difference image, which is an image of the difference information, on the display unit. In the difference image, there is a difference between the areas where the conversion was performed correctly and the areas where the conversion was not performed correctly. Therefore, by checking the difference image, the user can confirm whether or not the conversion was performed correctly. Furthermore, by checking the difference image, the user can also identify the areas where the conversion was not performed correctly.

[0022] In addition, when the input image contains irregular regions (such as diseased regions), it is difficult to appropriately perform the conversion of the image quality of the irregular regions. Therefore, the user can also appropriately grasp the irregular regions in the input image by checking the difference image.

[0023] A specific method for displaying the difference image on the display unit can be appropriately selected. For example, the display unit may display at least one of the input image and the converted image simultaneously (e.g., side by side) with the difference image. Also, the control unit may display the difference image by superimposing it on at least one of the input image and the converted image. In this case, the user can easily compare at least one of the input image and the converted image with the converted image. Further, the control unit may display the difference image alone on the display unit.

[0024] In the evaluation information acquisition step, the control unit may acquire evaluation information by inputting the input image and the converted image into a mathematical model trained by a machine learning algorithm (a mathematical model for evaluation information acquisition different from the mathematical model for converted image acquisition). In this case, even if the difference between the input image and the converted image is not acquired, the validity of the conversion can be appropriately evaluated.

[0025] In addition, the mode of the evaluation information output by the mathematical model for evaluation information acquisition can also be appropriately selected. For example, the mathematical model for evaluation information acquisition may output evaluation information indicating whether the conversion from the input image to the converted image is valid. In this case, it is easy to evaluate whether the conversion is valid. Also, the mathematical model for evaluation information acquisition may output information such as a numerical value indicating the degree of validity of the conversion as evaluation information.

[0026] Furthermore, it is possible to change the form of the evaluation information. For example, the control unit may acquire evaluation information using various parameters related to the image quality (at least the converted image). Parameters related to image quality may include, for example, the signal strength of the ophthalmic image, or an index indicating the quality of the signal (e.g., SSI (Signal Strength Index) or SQI (SLO Quality Index)), the ratio of the noise level to the signal level of the image (SNR (Signal to Noise Ratio)), the background noise level, the image contrast, etc. At least one of these may be used as parameters related to image quality. When an input image is converted to obtain a converted image with improved image quality from the input image, if the conversion is performed appropriately, the image quality of the converted image should be better than that of the input image. Therefore, for example, parameters indicating the image quality of the converted image may be acquired as evaluation information. Alternatively, the difference between the parameters indicating the image quality of the converted image and the parameters indicating the image quality of the input image may be acquired as evaluation information.

[0027] The control unit may further execute a warning step to warn the user if the conversion is evaluated as invalid based on the evaluation information acquired in the evaluation information acquisition step. In this case, the user can easily understand that the conversion from the input image to the converted image may not have been performed properly.

[0028] The specific method of warning processing can be selected as appropriate. For example, the control unit may warn the user by displaying at least one of a warning message and / or a warning image on the display unit. Alternatively, the control unit may warn the user by emitting at least one of a warning message and / or a warning sound from the speaker. Furthermore, the control unit may perform the warning processing while displaying the converted image on the display unit, or it may perform the warning processing without displaying the converted image.

[0029] If the control unit determines that the conversion is not valid based on the evaluation information obtained in the evaluation information acquisition step, it may further execute a display stop step to stop the display process of the converted image obtained in the converted image acquisition step on the display unit. In this case, the display of converted images that were not properly converted from the input image is suppressed. Therefore, the possibility that the user may not be able to accurately make various judgments based on the converted image is reduced.

[0030] Furthermore, it is possible to change how the evaluation information is used. For example, the control unit may display at least one of the acquired evaluation information, such as a numerical value or a graph, on the display unit. In this case, the user can easily understand whether the conversion from the input image to the converted image was performed appropriately based on the displayed evaluation information. Also, as mentioned above, the difference image between the input image and the converted image may be displayed on the display unit as evaluation information.

[0031] Furthermore, if the control unit evaluates the transformation as invalid based on the evaluation information, it may obtain the transformed image by inputting the input image into a different mathematical model than the one that performed the transformation evaluated as invalid. The characteristics of the transformation performed by the mathematical model differ depending on the algorithm and training data used to train the mathematical model. Therefore, if the transformation is evaluated as invalid, there is a possibility that the transformed image will be obtained appropriately by obtaining the transformed image using a different mathematical model.

[0032] As mentioned above, if the input image contains irregular areas (e.g., diseased areas), it is difficult to properly convert the image quality of these irregular areas. In this case, the difference image between the input image and the converted image will show the location of the irregular areas. Therefore, the control unit may display the difference image on the display unit regardless of the validity of the conversion from the input image to the converted image. As a result, the user can easily understand the location of the irregular areas based on the difference image.

[0033] In this case, the ophthalmic image processing program can be expressed as follows: An ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of tissue of an eye being examined, wherein the ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device, and the ophthalmic image processing program is characterized in that the ophthalmic image processing device executes the following steps: an image acquisition step of acquiring an ophthalmic image taken by an ophthalmic image capture device; a converted image acquisition step of acquiring a converted image by inputting the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm, thereby converting the image quality of the input image; a difference image acquisition step of acquiring a difference image, which is an image of the difference information of pixel values ​​between corresponding pixels of the input image input to the mathematical model and the converted image output from the mathematical model; and a difference image display step of displaying the difference image on a display unit.

[0034] The device that performs the image acquisition step, the converted image acquisition step, and the evaluation information acquisition step can be selected as appropriate. For example, the control unit of a personal computer (hereinafter referred to as "PC") may perform all of the converted image acquisition step and the evaluation information acquisition step. In other words, the control unit of the PC may acquire ophthalmic images from the ophthalmic imaging device and perform the converted image acquisition process based on the acquired ophthalmic images. Alternatively, the control unit of the ophthalmic imaging device may perform all of the converted image acquisition step and the evaluation information acquisition step. Furthermore, the control units of multiple devices (for example, an ophthalmic imaging device and a PC, etc.) may cooperate to perform the converted image acquisition step and the evaluation information acquisition step. [Brief explanation of the drawing]

[0035] [Figure 1] This is a block diagram showing the schematic configuration of the mathematical model building device 1, the ophthalmic image processing device 21, and the ophthalmic image acquisition devices 11A and 11B. [Figure 2]This figure shows an example of input and output training data when high-resolution two-dimensional tomographic images are used as converted images for a mathematical model. [Figure 3] This is a flowchart of the mathematical model construction process performed by the mathematical model construction device 1. [Figure 4] This is a flowchart of the ophthalmic image processing performed by the ophthalmic image processing device 21. [Figure 5] This figure shows an example of an ophthalmic image used as an input image. [Figure 6] This figure shows an example of a converted image obtained by converting the image quality of the input image shown in Figure 5. [Figure 7] This figure shows an example of a difference image between the input image shown in Figure 5 and the converted image shown in Figure 6. [Figure 8] This is an explanatory diagram illustrating an example of a method for evaluating the validity of the conversion from an input image to a converted image based on a difference image. [Figure 9] This is a flowchart for ophthalmic image processing in cases of deformity. [Modes for carrying out the invention]

[0036] (Device configuration) Hereinafter, one typical embodiment of the present disclosure will be described with reference to the drawings. As shown in Figure 1, in this embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image acquisition devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. The program that realizes the constructed mathematical model is stored in the storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 takes an ophthalmic image as an input image to the mathematical model and obtains a converted image in which the image quality of the input image has been transformed (in this embodiment, the image quality has been improved). The ophthalmic image processing device 21 also obtains evaluation information to evaluate the validity of the transformation of the converted image from the input image. The ophthalmic image acquisition devices 11A and 11B capture an ophthalmic image, which is an image of the tissue of the eye being examined.

[0037] As an example, a personal computer (hereinafter referred to as "PC") is used in the mathematical model construction device 1 of this embodiment. As will be described in detail later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using ophthalmic images acquired from the ophthalmic image acquisition device 11A (hereinafter referred to as "training ophthalmic images") and images obtained by converting the image quality of the training ophthalmic images. However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic image acquisition device 11A may function as the mathematical model construction device 1. In addition, the control units of multiple devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic image acquisition device 11A) may cooperate to construct the mathematical model.

[0038] Furthermore, this embodiment illustrates the case where a CPU is used as an example of a controller that performs various processing tasks. However, it goes without saying that controllers other than the CPU may be used in at least some of the various devices. For example, processing speed may be increased by adopting a GPU as the controller.

[0039] The mathematical model construction device 1 will now be described. The mathematical model construction device 1 is installed, for example, at a manufacturer that provides an ophthalmic image processing device 21 or an ophthalmic image processing program to a user. The mathematical model construction device 1 includes a control unit 2 that performs various control processing and a communication interface 5. The control unit 2 includes a CPU 3, which is a controller that manages the control, and a storage device 4 that can store programs and data. The storage device 4 stores a mathematical model construction program for executing the mathematical model construction processing (see Figure 3), which will be described later. The communication interface 5 connects the mathematical model construction device 1 to other devices (for example, an ophthalmic image acquisition device 11A and an ophthalmic image processing device 21, etc.).

[0040] The mathematical model building device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by the user to input various instructions to the mathematical model building device 1. The operation unit 7 can use at least one of the following: a keyboard, mouse, touch panel, etc. A microphone or the like may be used together with the operation unit 7, or in place of the operation unit 7, to input various instructions. The display device 8 displays various images. The display device 8 can use various devices capable of displaying images (for example, at least one of a monitor, display, projector, etc.). In this disclosure, "image" includes both still images and moving images.

[0041] The mathematical model building device 1 can acquire ophthalmic image data (hereinafter sometimes simply referred to as "ophthalmic images") from the ophthalmic image acquisition device 11A. The mathematical model building device 1 may acquire the ophthalmic image data from the ophthalmic image acquisition device 11A by at least one of the following: wired communication, wireless communication, or a removable storage medium (e.g., a USB memory stick).

[0042] The ophthalmic image processing device 21 will now be described. The ophthalmic image processing device 21 is installed, for example, in a facility that diagnoses or examines patients (e.g., a hospital or health checkup facility). The ophthalmic image processing device 21 is equipped with a control unit 22 that performs various control processing and a communication interface 25. The control unit 22 is equipped with a CPU 23, which is a controller that manages the control, and a storage device 24 that can store programs and data. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see Figures 4 and 9), which will be described later. The ophthalmic image processing program includes a program that realizes the mathematical model constructed by the mathematical model construction device 1. The communication interface 25 connects the ophthalmic image processing device 21 to other devices (e.g., an ophthalmic image acquisition device 11B and the mathematical model construction device 1).

[0043] The ophthalmic image processing device 21 is connected to the operation unit 27 and the display device 28. Various devices can be used in the operation unit 27 and the display device 28, as with the operation unit 7 and the display device 8 described above.

[0044] The ophthalmic image processing device 21 can acquire ophthalmic images from the ophthalmic image acquisition device 11B. The ophthalmic image processing device 21 may acquire ophthalmic images from the ophthalmic image acquisition device 11B by, for example, wired communication, wireless communication, or a removable storage medium (e.g., a USB memory). The ophthalmic image processing device 21 may also acquire programs for realizing mathematical models constructed by the mathematical model construction device 1 via communication or other means.

[0045] The ophthalmic image acquisition devices 11A and 11B will be described. As an example, in this embodiment, the case in which the ophthalmic image acquisition device 11A, which provides ophthalmic images to the mathematical model construction device 1, and the ophthalmic image acquisition device 11B, which provides ophthalmic images to the ophthalmic image processing device 21, are used will be described. However, the number of ophthalmic image acquisition devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from multiple ophthalmic image acquisition devices. Alternatively, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a single common ophthalmic image acquisition device.

[0046] In this embodiment, an OCT device is given as an example of the ophthalmic imaging device 11 (11A, 11B). However, other ophthalmic imaging devices (for example, a laser scanning optometry device (SLO), fundus camera, shineproof camera, or corneal endothelial cell imaging device (CEM), etc.) may be used.

[0047] The ophthalmic imaging device 11 (11A, 11B) comprises a control unit 12 (12A, 12B) that performs various control processing and an ophthalmic imaging unit 16 (16A, 16B). The control unit 12 comprises a CPU 13 (13A, 13B) which is a controller that manages the operation, and a storage device 14 (14A, 14B) that can store programs and data. When the ophthalmic imaging device 11 performs at least a part of the ophthalmic image processing described later (see Figures 4 and 9), it goes without saying that at least a part of the ophthalmic image processing program for performing the ophthalmic image processing is stored in the storage device 14.

[0048] The ophthalmic imaging unit 16 is equipped with various components necessary for capturing ophthalmic images of the eye under examination. The ophthalmic imaging unit 16 in this embodiment includes an OCT light source, a branching optical element that splits the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit for scanning the measurement light, an optical system for irradiating the eye under examination with the measurement light, and a light-receiving element that receives the combined light of the light reflected by the tissue of the eye under examination and the reference light.

[0049] The ophthalmic imaging device 11 can acquire two-dimensional and three-dimensional tomographic images of the fundus of the eye under examination. Specifically, the CPU 13 scans the scan lines with OCT light (measurement light) to acquire two-dimensional tomographic images (see Figure 5) of the cross-sections intersecting the scan lines. The CPU 13 can also acquire three-dimensional tomographic images of tissue by scanning the OCT light two-dimensionally. For example, the CPU 13 acquires multiple two-dimensional tomographic images by scanning the measurement light on each of multiple scan lines located at different positions within a two-dimensional region when viewing the tissue from the front. The CPU 13 then combines the acquired multiple two-dimensional tomographic images to acquire a three-dimensional tomographic image.

[0050] Furthermore, the CPU 13 can capture multiple ophthalmic images of the same area by scanning the same area of ​​tissue (in this embodiment, on the same scan line) multiple times with the measurement light. The CPU 13 can obtain an averaged image with suppressed speckle noise by performing an averaging process on multiple ophthalmic images of the same area. The image quality of the two-dimensional tomography can be improved by performing an averaging process on multiple two-dimensional tomography images of the same area. The averaging process may be performed, for example, by averaging the pixel values ​​of pixels at the same position among multiple ophthalmic images. The more images for which averaging is performed, the easier it is to suppress the effect of speckle noise, but the longer the acquisition time. The ophthalmic image acquisition device 11 performs a tracking process to make the scanning position of the OCT light follow the movement of the eye being examined while capturing multiple ophthalmic images of the same area.

[0051] (Mathematical model building process) Referring to Figures 2 and 3, the mathematical model construction process performed by the mathematical model construction device 1 will be described. The mathematical model construction process is executed by the CPU 3 according to the mathematical model construction program stored in the storage device 4.

[0052] In the mathematical model construction process, the mathematical model is trained using a training dataset, thereby constructing a mathematical model that outputs a transformed image with the image quality of the input image transformed. The training dataset includes input data (input training data) and output data (output training data). The mathematical model can transform various ophthalmic images into transformed images. The type of training dataset used to train the mathematical model is determined by the type of ophthalmic image whose image quality the mathematical model transforms. Below, we will explain the case where a two-dimensional tomographic image is input to the mathematical model, and the mathematical model outputs a two-dimensional tomographic image (high-resolution image) with improved image quality as the transformed image.

[0053] Figure 2 shows an example of input and output training data when a high-resolution two-dimensional tomographic image is output to a mathematical model as a transformed image. In the example shown in Figure 2, CPU3 acquires a set 40 of multiple two-dimensional tomographic images 400A to 400X taken from the same part of the tissue. CPU3 uses a portion of the multiple two-dimensional tomographic images 400A to 400X in set 40 (a number smaller than the number used for the averaging of the output training data described later) as input training data. CPU3 also acquires an averaged image 41 of the multiple two-dimensional tomographic images 400A to 400X in set 40 as output training data. When the mathematical model is trained with the input and output training data exemplified in Figure 2, the two-dimensional tomographic image is input to the trained mathematical model, and a high-resolution two-dimensional tomographic image with suppressed speckle noise is output as the transformed image.

[0054] Furthermore, the ophthalmic images used to convert image quality in the mathematical model are not limited to two-dimensional tomographic images of the fundus. For example, the ophthalmic image may be an image of a part of the eye other than the fundus. Also, the ophthalmic image may be a three-dimensional tomographic image, OCT angio image, or Enface image taken by an OCT device. An OCT angio image may be a two-dimensional frontal image of the fundus viewed from the front (i.e., in the direction of the line of sight of the eye being examined). An OCT angio image may be a motion contrast image obtained by processing at least two OCT signals acquired at different times for the same position. An Enface image is a two-dimensional frontal image of at least a portion of a three-dimensional tomographic image taken by an OCT device, viewed from a direction along the optical axis of the measurement light of the OCT device (frontal direction). Also, the ophthalmic image may be an image taken by a fundus camera, an image taken by a laser scanning ophthalmophotometer (SLO), or an image taken by a corneal endothelial cell imaging device.

[0055] Furthermore, it is possible to change the method for generating high-resolution ophthalmic images used as training data for output. For example, image quality may be improved by processing other than averaging.

[0056] Referring to Figure 3, the mathematical model construction process will be described. The CPU 3 acquires at least a portion of the ophthalmic images captured by the ophthalmic imaging device 11A as input training data (S1). In this embodiment, the ophthalmic image data is generated by the ophthalmic imaging device 11A and then acquired by the mathematical model construction device 1. However, the CPU 3 may acquire the ophthalmic image data by acquiring a signal (e.g., an OCT signal) that forms the basis for generating the ophthalmic image from the ophthalmic imaging device 11A and generating the ophthalmic image based on the acquired signal.

[0057] Next, CPU3 acquires output training data corresponding to the input training data acquired in S1 (S3). An example of the correspondence between input training data and output training data is as described above.

[0058] Next, CPU3 trains a mathematical model using a training dataset with a machine learning algorithm (S3). Commonly known machine learning algorithms include neural networks, random forests, boosting, and support vector machines (SVM).

[0059] Neural networks are techniques that mimic the behavior of biological nerve cell networks. Examples of neural networks include feedforward neural networks, RBF networks (radiating basis function networks), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Basian networks, etc.).

[0060] Random forests are a method for generating multiple decision trees by learning from randomly sampled training data. When using random forests, the system follows the branches of multiple decision trees that have been pre-trained as classifiers, and takes the average (or majority vote) of the results obtained from each decision tree.

[0061] Boosting is a technique for generating a strong classifier by combining multiple weak classifiers. It involves sequentially training simple, weak classifiers to construct a strong classifier.

[0062] SVM is a method for constructing a two-class pattern classifier using linear input elements. SVM learns the parameters of linear input elements based on a criterion (hyperplane separation theorem) that finds the margin-maximizing hyperplane that maximizes the distance to each data point from the training data.

[0063] A mathematical model refers to a data structure used to predict the relationship between input and output data. Mathematical models are built by being trained using a training dataset. As mentioned earlier, a training dataset is a set of input training data and output training data. For example, training updates the correlation data (e.g., weights) between each input and output.

[0064] In this embodiment, a multilayer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating the data to be predicted, and one or more hidden layers between the input and output layers. Each layer contains multiple nodes (also called units). Specifically, in this embodiment, a convolutional neural network (CNN), which is a type of multilayer neural network, is used. However, other machine learning algorithms may be used. For example, a generative adversarial network (GAN), which utilizes two competing neural networks, may be adopted as the machine learning algorithm.

[0065] Processes S1 to S3 are repeated until the construction of the mathematical model is complete (S5: NO). Once the construction of the mathematical model is complete (S5: YES), the mathematical model construction process ends. The program and data that realize the constructed mathematical model are incorporated into the ophthalmic image processing device 21.

[0066] (Ophthalmic image processing) An example of ophthalmic image processing performed by the ophthalmic image processing device 21 will be described with reference to Figures 4 to 8. Figures 4 to 8 illustrate the case in which the validity of image quality conversion is evaluated by a mathematical model based on the difference information (difference image) between the input image and the converted image. The ophthalmic image processing illustrated in Figure 4 is executed by the CPU 23 according to the ophthalmic image processing program stored in the storage device 24.

[0067] As shown in Figure 4, the CPU 23 acquires an ophthalmic image of the tissue of the eye being examined, which is captured by the ophthalmic imaging device (OCT device in this embodiment) 11B (S11). In S11 of this embodiment, a two-dimensional tomographic image of the fundus tissue of the eye being examined (see Figure 5) is acquired.

[0068] Next, the CPU 23 inputs the ophthalmic image acquired in S11 as an input image to a mathematical model trained by a machine learning algorithm, thereby obtaining a transformed image (in this embodiment, an improved image) in which the quality of the input image has been transformed (in this embodiment, the quality of the input image has been improved) (S12).

[0069] Figure 5 shows an example of an ophthalmic image used as an input image. Figure 6 shows a converted image obtained by transforming (improving the image quality of) the input image shown in Figure 5. The input image shown in Figure 5 has lower image quality than the converted image shown in Figure 6. However, the input image shown in Figure 5 is generated without performing averaging, or by performing averaging on a small number of ophthalmic images. Therefore, the input image shown in Figure 5 can be captured in a short time. When capturing an image of the same high quality as the converted image shown in Figure 6 using the ophthalmic image acquisition device 11B, it is necessary to capture images of the same area multiple times and perform averaging, making it difficult to shorten the capture time. In this embodiment, a high-quality converted image is obtained by inputting an input image captured in a short time into a mathematical model. Therefore, a high-quality image is obtained while suppressing the length of the capture time.

[0070] Next, the CPU 23 acquires the difference information of pixel values ​​between corresponding pixels in the input image input to the mathematical model in S12 (see Figure 5) and the transformed image output from the mathematical model in S12 (see Figure 6) as evaluation information (S13). Evaluation information is information used to evaluate the validity of the transformation from the input image to the transformed image by the mathematical model. If the image quality of the input image is appropriately transformed and the transformed image is output, the difference between the input image and the transformed image will be small. On the other hand, depending on the state of the input image, the image quality transformation may not be performed appropriately. For example, if there are areas in the input image that were included in a low proportion in the training dataset (ophthalmic images) used to train the mathematical model (for example, irregular areas such as lesion areas), the transformation of the irregular areas will be difficult to perform appropriately. Therefore, if irregular areas in the input image affect the transformation and the transformation of the input image is not performed appropriately, the difference between the input image and the transformed image will be large. Thus, by acquiring the difference information as evaluation information, the validity of the transformation from the input image to the transformed image can be appropriately evaluated.

[0071] The CPU 23 may also obtain difference information after performing a Fourier transform (e.g., a two-dimensional Fourier transform) on both the input image and the transformed image. Periodic artifacts may occur in the transformed image. When periodic artifacts occur in the transformed image, the input image and the transformed image will exhibit different frequency distributions. Therefore, using the Fourier transform allows for a more appropriate evaluation of whether or not periodic artifacts occur.

[0072] In step S13 of this embodiment, a difference image (see Figure 7) showing the distribution of difference values ​​acquired for each pixel is acquired as difference information. In the difference image illustrated in Figure 7, the gray areas where the brightness is an intermediate value (for example, 128, which is the intermediate value when the brightness changes in the range of 1 to 256) are displayed as pixels with small difference values. In the difference image, there is a difference between the areas where the conversion was performed appropriately and the areas where the conversion was not performed appropriately. Therefore, the validity of the conversion from the input image to the converted image is appropriately evaluated based on the difference image.

[0073] Note that the difference information may be information other than the difference image. For example, in S13, the average value of the difference between multiple pixels may be obtained as the difference information. Also, the difference information may be the difference in pixel values ​​between corresponding pixels, or it may be the ratio of one pixel value to the other pixel value.

[0074] Next, the CPU 23 evaluates the validity of the conversion from the input image to the converted image in S12 based on the difference image acquired in S13 (see Figure 7) (S14). Specifically, in S14 of this embodiment, it is evaluated whether or not the conversion in S12 was valid.

[0075] Referring to Figure 8, an example of a method for evaluating the validity of a conversion based on a difference image will be described. As shown in Figure 8, even when the conversion from the input image to the converted image is performed appropriately, pixels 51 with large difference values ​​are scattered within the image region 50 of the difference image. On the other hand, in regions where the conversion was not performed appropriately due to the presence of irregular areas (e.g., diseased areas), pixels 51 with large difference values ​​are densely clustered. Therefore, in this embodiment, the CPU 23 evaluates that there are regions where the conversion was not performed appropriately (i.e., the conversion is not valid) if the difference value in any region containing multiple pixels is greater than or equal to a threshold. In the example shown in Figure 8, the difference value in region 55 is greater than or equal to the threshold. Therefore, the CPU 23 evaluates region 55 as a region where the conversion was not performed appropriately (a region where irregular areas exist).

[0076] Furthermore, the CPU 23 performs a smoothing process on the pixel values ​​of the difference image (the difference value corresponding to each pixel), and then evaluates the validity of the conversion based on the difference image. Therefore, even if the conversion is performed appropriately, the influence of the scattered pixels 51 is suppressed, and the validity of the conversion is evaluated more appropriately.

[0077] The specific method for evaluating the validity of the conversion based on the difference values ​​within a region can be selected as appropriate. For example, in this embodiment, the CPU 23 evaluates that the image quality conversion in a region is not valid if the average value of the difference values ​​within that region is greater than or equal to a threshold. However, the CPU 23 may also evaluate whether the conversion is valid based on the number of pixels within a unit region whose difference values ​​are greater than or equal to a threshold.

[0078] Alternatively, instead of evaluating whether the conversion in S12 was valid, the CPU 23 may obtain information indicating the degree of validity of the conversion in S12 (e.g., numerical values ​​or graphs) based on the difference information. The CPU 23 may also notify the user of the information indicating the degree of validity of the conversion as evaluation information (e.g., displayed on the display device 28).

[0079] Returning to the explanation of Figure 4, if the CPU 23 determines that the conversion from the input image to the converted image in S12 is valid (S15: YES), it displays the converted image acquired in S12 on the display device 28 (S16). On the other hand, if the CPU 23 determines that the conversion from the input image to the converted image in S12 is not valid (S15: NO), it stops the process of displaying the converted image acquired in S12 on the display device 28 (i.e., it does not execute the process in S16).

[0080] Furthermore, if the CPU 23 evaluates that the conversion is not valid (S15:NO), it executes a warning process to the user (S17). For example, the CPU 23 warns the user by displaying a warning message such as "Image conversion was not performed properly" or a warning image on the display device 28. However, the method of warning can be changed as appropriate. For example, the CPU 23 may warn the user by emitting at least one of a warning message and / or a warning sound from the speaker.

[0081] Furthermore, the CPU 23 can also perform a warning process while displaying the converted image that was not converted properly on the display device 28. In this case, the CPU 23 may use a warning message such as "The displayed converted image may be inappropriate" in the process of S17. Also, if the CPU 23 evaluates that the conversion is not valid (S15: NO), it may stop the process of displaying the converted image acquired in S12 on the display device 28 and instead display the ophthalmic image used as the input image on the display device 28. In this case, the user can observe the desired area based on the ophthalmic image before the image quality was converted.

[0082] Next, the CPU 23 displays the difference image acquired in S13 (see Figure 7) on the display device 28 (S18). As mentioned above, the difference image shows a difference between areas where the conversion was performed correctly and areas where the conversion was not performed correctly. Therefore, the user can check whether the conversion was performed correctly by checking the difference image. Furthermore, the user can also identify areas where the conversion was not performed correctly by checking the difference image. In addition, if the input image contains irregular areas (e.g., diseased areas), the image quality conversion of the irregular areas is unlikely to be performed correctly. Therefore, the user can also identify irregular areas in the input image by checking the difference image.

[0083] (Example of transformation) An example of a modification of the above embodiment will be described with reference to Figure 9. Figure 9 is a flowchart of ophthalmic image processing in the modification example. In the modification example shown in Figure 9, the similarity (e.g., correlation, etc.) between the difference image and the input image is obtained as evaluation information, and the validity of the conversion is evaluated based on the similarity. At least a part of the ophthalmic image processing exemplified in the above embodiment (see Figure 4) can also be similarly adopted in the ophthalmic image processing of the modification example shown in Figure 9. Therefore, for processes that can be performed in the same way as in the above embodiment, the same step numbers as in the above embodiment are assigned, and their explanations are omitted or simplified.

[0084] In the ophthalmic image processing of the transformation example shown in Figure 9, the CPU 23 obtains a difference image (see Figure 7) between the input image and the transformed image (see Figure 7) after executing the transformation image acquisition process (S12) (S23). Next, the CPU 23 obtains the similarity between the input image and the difference image as evaluation information (S24). As mentioned above, if the image quality of the input image is appropriately transformed, the difference between the input image and the transformed image becomes small, and therefore the similarity between the difference image and the input image becomes small. On the other hand, if the transformation of the input image is not performed appropriately and irregular parts in the input image affect the transformation, the location (region) of the irregular parts in the input image and the location (region) where the difference value is large in the difference image become similar, so the similarity between the input image and the difference image becomes large. Therefore, by obtaining the similarity between the difference image and the input image as evaluation information, the validity of the transformation is appropriately evaluated.

[0085] Next, the CPU 23 evaluates the validity of the conversion from the input image to the converted image in S12 based on the similarity obtained in S24 (S25). Specifically, in S25 of this embodiment, whether or not the conversion in S12 was valid is evaluated based on the similarity. As mentioned above, if the conversion is performed appropriately, the similarity between the input image and the difference image will decrease. On the other hand, if the conversion is not performed appropriately, the similarity between the input image and the difference image will increase. Therefore, the CPU 23 can appropriately evaluate whether or not the conversion from the input image to the converted image was valid by determining whether or not the value indicating similarity is above a threshold.

[0086] Various values ​​(e.g., correlation coefficient) can be used as appropriate to indicate the degree of similarity. Alternatively, instead of the CPU 23 evaluating whether the conversion in S12 was valid, it may display information indicating the degree of validity of the conversion in S12 (e.g., numerical values, correlation diagrams, or graphs) on the display device 28 as evaluation information.

[0087] The technologies disclosed in the above embodiments and modification examples are merely examples. Therefore, it is possible to modify the technologies exemplified in the above embodiments and modification examples. First, in the above embodiments, the validity of the conversion from the input image to the converted image is evaluated based on difference information (difference image). In the above modification examples, the validity of the conversion is evaluated based on the similarity between the input image and the difference image. However, the method for obtaining evaluation information to evaluate the validity of the conversion is not limited to the methods exemplified in the above embodiments and modification examples.

[0088] For example, CPU23 may acquire evaluation information using a mathematical model trained by a machine learning algorithm. In this case, the mathematical model (a mathematical model for acquiring evaluation information) may be pre-trained, for example, using input images and transformed images as input training data, and evaluation information indicating the validity of the transformation between the input images and transformed images in the input training data as output training data. The output training data may be generated by the user comparing the input images and transformed images. CPU23 may acquire the evaluation information output by the mathematical model for acquiring evaluation information by inputting the input images and transformed images into the mathematical model for acquiring evaluation information. By acquiring evaluation information through the mathematical model for acquiring evaluation information, the validity of the transformation can be appropriately evaluated even if differences between the input images and transformed images are not acquired.

[0089] Furthermore, the mathematical model for acquiring evaluation information may output evaluation information indicating whether the conversion from the input image to the converted image is valid, or it may output evaluation information such as a numerical value indicating the degree of validity of the conversion.

[0090] Furthermore, the process executed when it is determined that the conversion from the input image to the converted image is not valid can also be modified as appropriate. For example, if CPU23 evaluates the conversion as not valid based on the evaluation information, it may obtain the converted image by inputting the input image into a different mathematical model than the one that performed the conversion that was evaluated as not valid. The characteristics of the conversion performed by the mathematical model differ depending on the algorithm and training data used to train the mathematical model. Therefore, if the conversion is evaluated as not valid, there is a possibility that the converted image will be obtained appropriately by obtaining the converted image using a different mathematical model.

[0091] It is also possible to implement only some of the techniques exemplified in the above embodiments and variations. For example, in the ophthalmic image processing shown in Figure 4, if the conversion is evaluated as unsuitable (S15:NO), both the process of stopping the display of the converted image and the warning process are executed. However, it is also possible to omit at least one of the processes of stopping the display of the converted image and the warning process.

[0092] The process of acquiring ophthalmic images in S11 of Figures 4 and 9 is an example of the "image acquisition step". The process of acquiring converted images in S12 of Figures 4 and 9 is an example of the "converted image acquisition step". The process of acquiring evaluation information in S13 of Figure 4 and S24 of Figure 9 is an example of the "evaluation information acquisition step". The process of evaluating the validity of the conversion in S14 of Figure 4 is an example of the "first evaluation step". The process of evaluating the validity of the conversion in S25 of Figure 9 is an example of the "second evaluation step". The process of displaying the difference image in S18 of Figures 4 and 9 is an example of the "difference image display step". The warning process shown in S17 of Figures 4 and 9 is an example of the "warning step". The process of stopping the display of the converted image in S15:NO of Figures 4 and 9 is an example of the "display stop step". [Explanation of Symbols]

[0093] 11A, 11B Ophthalmic imaging device 21 Ophthalmic Image Processing Equipment 23 CPU 24 Storage device 28 indicates the device

Claims

1. An ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of tissue of the eye being examined, The ophthalmic image processing program is executed by the control unit of the ophthalmic image processing device, An image acquisition step in which ophthalmic images taken by an ophthalmic imaging device are acquired, A transformed image acquisition step involves inputting the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm, thereby acquiring a transformed image with improved image quality by suppressing the effects of speckle noise in the input image. A difference information acquisition step that acquires difference information of pixel values ​​between corresponding pixels in the input image input to the mathematical model and the transformed image output from the mathematical model, A difference image display step involves displaying a difference image, which is an image of the distribution of the difference information, on a display unit. The steps include: displaying information on the display unit indicating the degree of validity of the transformation from the input image to the transformed image by the mathematical model, based on the similarity between the difference image and the input image; An ophthalmic image processing program characterized in that it is executed by the ophthalmic image processing device.

2. An ophthalmic image processing program according to claim 1, A second evaluation step in which, if the value indicating the similarity is greater than or equal to a threshold, the conversion from the input image to the converted image is evaluated as not being valid. An ophthalmic image processing program characterized in that it is executed by the ophthalmic image processing device.

3. An ophthalmic image processing device that processes ophthalmic images, which are images of the tissue of the eye being examined, The control unit of the ophthalmic image processing device is An image acquisition step in which ophthalmic images taken by an ophthalmic imaging device are acquired, A transformed image acquisition step involves inputting the ophthalmic image acquired in the image acquisition step as an input image to a mathematical model trained by a machine learning algorithm, thereby acquiring a transformed image with improved image quality by suppressing the effects of speckle noise in the input image. A difference information acquisition step that acquires difference information of pixel values ​​between corresponding pixels in the input image input to the mathematical model and the transformed image output from the mathematical model, A difference image display step involves displaying a difference image, which is an image of the distribution of the difference information, on a display unit. The steps include: displaying information on the display unit indicating the degree of validity of the transformation from the input image to the transformed image by the mathematical model, based on the similarity between the difference image and the input image; An ophthalmic image processing apparatus characterized by performing the following:

Images