Ophthalmologic image processing program and ophthalmologic image processing device

Abstract

To provide an ophthalmologic image processing program and an ophthalmologic image processing device capable of causing a user to take necessary measures efficiently and appropriately after causing the user to appropriately determine the quality of an acquired image.SOLUTION: A control part acquires an ophthalmologic image captured by an ophthalmologic image capturing device. The control part causes a display part to display a confirmation screen 30 for causing a user to determine the quality of the ophthalmologic image captured by the ophthalmologic image capturing device. The control part acquires a high-quality image 42 in which the quality of an input image 41 is improved by inputting the ophthalmologic image captured by the ophthalmologic image capturing device to a mathematic model trained by a machine learning algorithm. The control part acquires similarity information, which is the information on the similarity of the same position, of the input image 41 and the high-quality image 42. The control part causes the similarity information to be displayed on the confirmation screen 30.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used to process ophthalmic images of a subject's eye. [Background technology]

[0002] Techniques have been proposed for acquiring various medical information using mathematical models trained by machine learning algorithms. For example, in an ophthalmologic image processing device described in Patent Document 1, an ophthalmologic image is input as an input image to a mathematical model trained by a machine learning algorithm, and a converted image is acquired by converting the image quality of the input image. Furthermore, a difference image is acquired by visualizing the difference information of pixel values ​​between corresponding pixels in the input image and the converted image. The ophthalmologic image processing device described in Patent Document 1 attempts to evaluate the validity of image quality conversion using a mathematical model based on the similarity between the input image and the difference image. [Prior art documents] [Non-patent literature]

[0003] [Patent Document 1] International Publication No. 2021 / 045019 Summary of the Invention [Problem to be solved by the invention]

[0004] There may be cases where an image acquired as a high-quality image using a mathematical model is not good. Furthermore, there may be cases where the input image itself is not good. If an image that is not good is used in medical treatment as is, it may lead to a decline in the quality of medical treatment. Therefore, it is desirable for a user (e.g., a medical professional) to be able to determine the quality of the acquired image and then efficiently and appropriately take measures such as recapturing the image. Patent Document 1 discloses a technology that attempts to evaluate the appropriateness of image quality conversion based on a difference image, but it is difficult to say that it discloses a technology that allows a user to appropriately take measures such as recapturing the image.

[0005] A typical object of the present disclosure is to provide an ophthalmic image processing program and an ophthalmic image processing device that enable a user to appropriately determine the quality of an acquired image and then take necessary measures efficiently and appropriately. [Means for solving the problem]

[0006] An ophthalmic image processing program provided by a typical embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of the tissues of a test eye.When the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device, the ophthalmic image processing device executes the following steps: an image acquisition step that acquires an ophthalmic image captured by the ophthalmic image capturing device; a confirmation screen display step that displays a confirmation screen on a display unit to allow a user to confirm the quality of the ophthalmic image captured by the ophthalmic image capturing device; a high-quality image acquisition step that acquires a high-quality image with improved quality by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm; a similarity information acquisition step that acquires similarity information, which is information regarding the similarity between the input image and the high-quality image at the same position; and a similarity information display step that displays the similarity information on the confirmation screen.

[0007] An ophthalmic image processing device provided by a typical embodiment of the present disclosure is an ophthalmic image processing device that processes ophthalmic images, which are images of the tissues of a test eye, and a control unit of the ophthalmic image processing device executes an image acquisition step of acquiring an ophthalmic image captured by an ophthalmic image capturing device; a confirmation screen display step of displaying a confirmation screen on a display unit to allow a user to confirm the quality of the ophthalmic image captured by the ophthalmic image capturing device; a high-quality image acquisition step of acquiring a high-quality image with improved quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm; a similarity information acquisition step of acquiring similarity information, which is information regarding the similarity between the input image and the high-quality image at the same position; and a superimposition display step of displaying the similarity information on the confirmation screen.

[0008] According to the ophthalmological image processing program and ophthalmological image processing device according to the present disclosure, the quality of the acquired image can be appropriately determined by the user, and necessary measures can be easily taken efficiently and appropriately. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmologic image processing device 21, and an ophthalmologic image capturing device 11. FIG. [Figure 2] 10A and 10B are diagrams illustrating an example of input training data and output training data when a high-quality motion contrast image is output to a mathematical model. [Figure 3] 10 is a flowchart of ophthalmologic image processing executed by the ophthalmologic image processing device 21. [Figure 4] FIG. 10 is a diagram showing an example of a confirmation screen 30. [Figure 5] 1 is an explanatory diagram for explaining a process of generating a similarity image 43 from an input image 41 and a high-quality image 42. FIG. [Figure 6] FIG. 5 is a diagram showing an example of a state in which a re-photograph area 50 is set on the confirmation screen 30 shown in FIG. 4. [Figure 7]10 is an explanatory diagram for explaining an example of a method for replacing a re-photographed area 50 of an original input image 41O with a re-photographed image 41R. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Summary> The control unit of the ophthalmic image processing device exemplified in the present disclosure executes an image acquisition step, a confirmation screen display step, a converted image acquisition step, a similarity information acquisition step, and a similarity information display step. In the image acquisition step, the control unit acquires an ophthalmic image captured by the ophthalmic image capturing device. In the confirmation screen display step, the control unit displays a confirmation screen on the display unit to allow a user to confirm the quality of the ophthalmic image captured by the ophthalmic image capturing device. In the converted image acquisition step, the control unit inputs the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm, thereby acquiring a high-quality image with improved quality of the input image. In the similarity information acquisition step, the control unit acquires similarity information, which is information regarding the similarity between the input image and the high-quality image at the same position (e.g., similarity of image information between regions or pixels at the same position). In the similarity information display step, the control unit displays the similarity information on the confirmation screen.

[0011] In areas where the accuracy of image quality conversion (improvement of image quality) by the mathematical model is high, the similarity between the input image and the high-quality image tends to be high. On the other hand, in areas where the accuracy of image quality conversion by the mathematical model is low, the similarity between the input image and the high-quality image tends to be low. Therefore, the user can appropriately grasp the accuracy of the image quality conversion by the mathematical model from the similarity information. Furthermore, the similarity information is displayed on a confirmation screen that allows the user to confirm the quality of the ophthalmic image. Therefore, the user can grasp the accuracy of the image quality conversion by the mathematical model when confirming the quality of the image after capturing the input image. Therefore, the user can appropriately determine the quality of the captured or acquired ophthalmic image and then efficiently and appropriately take measures such as re-capturing the image.

[0012] The image quality converted by the mathematical model can be selected as appropriate. For example, the control unit may use a mathematical model to acquire a converted image in which at least one of the noise amount, contrast, resolution, etc. of the input image is converted.

[0013] This disclosure illustrates a case in which an ophthalmic imaging device functions as an ophthalmic image processing device that processes ophthalmic images. In this case, the ophthalmic imaging device can perform processes such as displaying a confirmation screen on a display unit while capturing an ophthalmic image, acquiring a high-quality image of the captured ophthalmic image, acquiring similarity information, and displaying the similarity information on the confirmation screen. This allows a user to determine the quality of an ophthalmic image immediately after capturing it, making it easier to efficiently take measures such as recapturing the image. However, devices that can function as ophthalmic image processing devices are not limited to ophthalmic imaging devices. For example, a personal computer (hereinafter referred to as a "PC") may be used as the ophthalmic image processing device. In this case, the PC may acquire data of ophthalmic images captured by the ophthalmic imaging device via at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), and the like, and perform processing on the acquired ophthalmic images. Furthermore, control units of multiple devices may cooperate to process ophthalmic images.

[0014] The ophthalmic image may be a motion contrast image (e.g., an OCT angiogram) obtained by processing at least two OCT signals acquired at different times for the same position using an OCT device, which is an ophthalmic image capturing device.

[0015] A motion contrast image shows information about the movement of the subject. Therefore, a motion contrast image can easily reveal, for example, blood vessels through which blood flows in biological tissue. On the other hand, if a motion contrast image is captured to reveal blood vessels, and biological tissue other than blood vessels moves during the image capture, the effects of the biological tissue movement will also appear in the motion contrast image. In this case, the quality of at least one of the input image and the high-quality image will be degraded. In addition, various image parameters (e.g., the signal strength of an ophthalmic image or an index indicating signal quality (e.g., SSI (Signal Strength Index) or SQI (SLO Quality Index)), the ratio of the image's signal level to the noise level (SNR (Signal to Noise Ratio)), background noise level, image contrast, etc.) also affect the quality of a motion contrast image. However, even if a user looks at the input image and the high-resolution image, it is often difficult to determine whether the quality has deteriorated. For example, if the accuracy of image conversion using a mathematical model is low, false blood vessels that did not appear in the input image may appear in the high-resolution image. However, even if a user looks at only the high-resolution image, it is difficult to determine whether the areas that appear to be blood vessels in the high-resolution image are the original blood vessels that appeared in the input image or false blood vessels that appeared due to image conversion.

[0016] In contrast, when a high-quality image is obtained by inputting a motion contrast image as an input image into a mathematical model, the accuracy of image quality conversion by the mathematical model is greatly affected by the quality of the motion contrast image input into the mathematical model. Therefore, by obtaining a high-quality image using a motion contrast image as an input image and displaying similarity information between the input image and the high-quality image on a confirmation screen, it becomes easier to appropriately grasp the quality of the ophthalmic image.

[0017] The motion contrast image input as an input image to the mathematical model may be a two-dimensional front image of the biological tissue viewed from the front along the optical axis of the OCT measurement light. For example, the motion contrast image as an input image may be a two-dimensional enface image of the imaging area of ​​a three-dimensional image viewed from the front. The enface image data may be, for example, integrated image data in which brightness values ​​are integrated in the depth direction (Z direction) at each position in the direction intersecting the optical axis of the OCT measurement light (XY direction), integrated values ​​of spectral data at each position in the XY direction, brightness data at each position in the XY direction in a certain depth direction, brightness data at each position in the XY direction in any layer of the biological tissue (e.g., the retinal surface layer, etc.). The motion contrast image as an input image may also be a three-dimensional image.

[0018] However, the input image is not limited to a motion contrast image. For example, when an OCT device is used as the ophthalmologic imaging device, the input image may be a two-dimensional tomographic image or a three-dimensional tomographic image of tissue. Furthermore, the input image may be captured by a device other than the OCT device (for example, at least one of a fundus camera, a scanning laser ophthalmoscope (SLO), an infrared camera, etc.).

[0019] In the similarity information acquisition step, the control unit may generate and acquire, as similarity information, a similarity image that visualizes the similarity at the same position between the input image and the high-quality image. As described above, in areas where the accuracy of image quality conversion (image quality improvement) by the mathematical model is high, the similarity between the input image and the high-quality image is likely to be high. On the other hand, in areas where the accuracy of image quality conversion by the mathematical model is low, the similarity between the input image and the high-quality image is likely to be low. Therefore, the user can appropriately grasp the distribution of the accuracy of image quality conversion by the mathematical model from the similarity image displayed on the confirmation screen, and then efficiently and appropriately take measures such as re-capturing the image.

[0020] A specific method for generating the similarity image can be selected as appropriate. For example, the control unit may normalize the similarity of image information (e.g., pixel values) between regions or pixels at the same position in the input image and the high-quality image, and represent each pixel of the similarity image based on the normalized value. The control unit may also acquire difference information (e.g., pixel value differences) between regions or pixels at the same position in the input image and the high-quality image. The smaller the difference indicated by the difference information, the higher the similarity. Therefore, the control unit may represent each pixel of the similarity image based on the difference information. It is also possible to acquire, as the similarity, at least one of the mean absolute error, mean square error, and correlation coefficient of pixel values ​​within a local region including a pixel of interest. The control unit may also generate the similarity image after performing a Fourier transform (e.g., a two-dimensional Fourier transform) on each of the input image and the high-quality image. Periodic artifacts may occur in the high-quality image. When periodic artifacts occur in the high-quality image, the input image and the high-quality image exhibit different frequency distributions. Therefore, by using the Fourier transform, the occurrence or non-occurrence of periodic artifacts can be more appropriately evaluated.

[0021] In the similarity information display step, a superimposed image in which the similarity image is superimposed on at least one of the input image and the high-quality image may be displayed on the confirmation screen. As described above, the similarity image makes it easy to understand the distribution of accuracy of image quality conversion using the mathematical model. Furthermore, by superimposing the similarity image on at least one of the input image and the high-quality image, the user can more appropriately understand the distribution of accuracy of image quality conversion using the mathematical model on the ophthalmological image on which the similarity image is superimposed. Therefore, the user can more efficiently and appropriately take measures such as re-capturing the image.

[0022] In the similarity information display step, the control unit may increase the transparency of the pixels that make up the similarity image as the similarity between the pixel values ​​of the input image and the high-resolution image decreases, and superimpose the similarity image on at least one of the input image and the high-resolution image.

[0023] In this case, in an area where the similarity between the input image and the high-quality image is low, the user can easily grasp the state of the ophthalmic image (input image or high-quality image) on which the similarity image is superimposed. Therefore, the user can easily properly grasp the state of an area where the accuracy of image quality conversion was low in the ophthalmic image on which the similarity image is superimposed.

[0024] However, the specific method for superimposing and displaying the similarity image may be changed. For example, the control unit may superimpose and display the similarity image on at least one of the input image and the high-quality image by increasing the transparency of the pixels constituting the similarity image as the similarity between the pixel values ​​of the input image and the high-quality image increases. In this case, the user can easily grasp, for example, the state of an area in the ophthalmological image on which the similarity image is superimposed, where the accuracy of image quality conversion was high. Furthermore, the control unit may change the color of each pixel constituting the similarity image according to the similarity between the pixel values ​​of the input image and the high-quality image. In this case, the user can more easily grasp the distribution of image quality conversion accuracy.

[0025] The ophthalmologic image processing device may further execute a re-photographing instruction receiving step and a re-photographing step. In the re-photographing instruction receiving step, the control unit receives an input of an instruction to re-photograph the ophthalmologic image while displaying the superimposed image on the confirmation screen. In the re-photographing step, the control unit executes re-photographing of the ophthalmologic image by the ophthalmologic image capturing device when the instruction to re-photograph is input by the user in the re-photographing instruction receiving step.

[0026] In this case, the user can properly grasp the distribution of accuracy of image quality conversion by the mathematical model on the ophthalmologic image on which the similarity image is superimposed, and if the accuracy of the image quality conversion is low, the user can immediately re-capture the ophthalmologic image, which makes it easier to efficiently and appropriately handle the re-capture of the image.

[0027] In the re-photographing step, when an instruction to perform re-photographing is input by the user, re-photographing of an ophthalmologic image may be performed for a tissue range corresponding to a part of the re-photographed area within the image area of ​​the superimposed image.

[0028] In this case, for example, it is possible to re-capture a part of the image area of ​​the superimposed image that has a low similarity indicated by the similarity image as a re-capture area. Therefore, the time required for re-capturing can be shortened compared to when an ophthalmologic image is re-captured within a tissue range corresponding to the entire image area of ​​the superimposed image. Therefore, it is easier to handle image re-capture more efficiently and appropriately.

[0029] When re-photographing of an ophthalmic image of a tissue range corresponding to a portion of a re-photographed area is performed in the re-photographing step, the ophthalmic image processing device may further perform a replacement step of replacing an image within the re-photographed area in the ophthalmic image before the re-photographing is performed with the re-photographed ophthalmic image.

[0030] In this case, an image in the re-photographed area (e.g., an image in the area where the similarity indicated by the similarity image was low) of the ophthalmic image before the re-photographing is performed is replaced with the re-photographed ophthalmic image. That is, the ophthalmic image re-photographed in a short time is appropriately combined with the ophthalmic image before the re-photographing is performed. Therefore, an appropriate ophthalmic image is efficiently acquired.

[0031] The ophthalmologic image processing device may further execute an area receiving step of receiving an input of an instruction to specify a re-photographing area for re-photographing the ophthalmologic image on the superimposed image displayed on the confirmation screen. In the re-photographing step, the control unit may execute re-photographing of the ophthalmologic image for a tissue range corresponding to the re-photographing area specified in the area receiving step, among the image areas of the superimposed image.

[0032] In this case, the user can grasp the distribution of accuracy of image quality conversion by the mathematical model from the superimposed image on which the similarity image is superimposed, and then easily and appropriately specify the re-capture area on the superimposed image. Therefore, the re-capture area can be easily set appropriately.

[0033] However, the method for setting the re-capture area may be changed. For example, the control unit may automatically set at least a part of the image area of ​​the superimposed image where the similarity indicated by the similarity image is equal to or less than a threshold as the re-capture area. Alternatively, the control unit may re-capture an ophthalmologic image for a tissue range corresponding to the entire image area of ​​the superimposed image. This also makes it easier to efficiently acquire an appropriate ophthalmologic image.

[0034] The control unit may also determine whether to re-capture the ophthalmic image using the ophthalmic image capturing device based on the similarity between the input image and the high-quality image (e.g., an average value of the similarity in at least a portion of the overlapping image area). If the control unit determines to re-capture the ophthalmic image, it may re-capture the ophthalmic image. In this case, the re-capture of the ophthalmic image is automatically and appropriately performed based on the similarity indicating the accuracy of the image quality conversion.

[0035] Furthermore, after the similarity image is generated and acquired in the similarity information acquisition step, the control unit may display the acquired similarity image together with or superimposed on an observation image of the tissue of the subject's eye (e.g., a two-dimensional moving image of the tissue of the subject's eye captured from the front) displayed on the display unit for re-capturing the ophthalmic image. In this case, the user can appropriately determine whether to perform re-capture based on the observation image while understanding the distribution of image transformation accuracy from the similarity image. For example, the control unit may accept input of an instruction to specify a re-capture area on a superimposed image of the observation image and the similarity image. In this case, the user can appropriately specify a region requiring re-capture while understanding the distribution of image transformation accuracy from the superimposed image.

[0036] The control unit may switch whether to superimpose the similarity image on at least one of the input image and the high-quality image in response to an instruction input by the user. In this case, the user can properly understand both the ophthalmological image with the similarity image superimposed and the ophthalmological image without the similarity image superimposed. This makes it easier to properly understand the quality of the ophthalmological image.

[0037] In the similarity information display step, the control unit may display a superimposed image in which the similarity image is superimposed on the high-quality image on the confirmation screen. The accuracy of image quality conversion using a mathematical model may be reduced by various influences (e.g., the influence of tissue movement during imaging, the influence of imaging conditions, the influence of image parameters, etc.). A similarity image that visualizes the similarity between the input image and the high-quality image tends to clearly show the distribution of areas where the accuracy of image quality conversion using the mathematical model is low and areas where it is high. Therefore, by superimposing the similarity image on the high-quality image, the user can properly grasp the presence or absence and distribution of areas in the high-quality image where the accuracy of image quality conversion is low.

[0038] In the similarity information display step, the control unit may display a superimposed image, in which the similarity image is superimposed on the input image, on the confirmation screen. If an area of ​​poor quality (hereinafter referred to as a "bad area") exists in at least a portion of the input image, when the image quality of the input image is improved using a mathematical model, the accuracy of image quality conversion in the bad area is likely to be lower than the accuracy of image quality conversion in an area of ​​good quality (hereinafter referred to as a "good area"). Therefore, in an area where the similarity indicated by the similarity image is low (i.e., an area where the accuracy of image quality conversion is low), the quality of the input image input to the mathematical model is likely to be poor. Therefore, by superimposing the similarity image on the input image, the user can properly grasp the presence or absence and distribution of bad areas in the input image.

[0039] However, the control unit may acquire similarity information other than the similarity image (e.g., an average value of similarities or a similarity level in the entire image area between the input image and the high-quality image) and display the acquired similarity information on the confirmation screen. Even in this case, the user can appropriately determine the quality of the captured or acquired ophthalmic image and efficiently and appropriately take measures such as re-capturing the image. Furthermore, the control unit may execute the re-capture instruction receiving step and the re-capture step while displaying similarity information other than the similarity image on the confirmation screen. Even in this case, the user can efficiently and appropriately take measures such as re-capturing the image after understanding the accuracy of the image quality conversion using the mathematical model. Moreover, when the control unit generates and acquires a similarity image as similarity information, it may display the similarity image on the confirmation screen without superimposing it on the ophthalmic image. Even in this case, the user can appropriately understand the distribution of the accuracy of the image quality conversion using the mathematical model from the similarity image displayed on the confirmation screen and efficiently and appropriately take measures such as re-capturing the image.

[0040] <Embodiment> (Device configuration) A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, this embodiment uses a mathematical model construction device 1, an ophthalmic image processing device 21, and an ophthalmic image capturing device 11. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. A program for realizing the constructed mathematical model is stored in a storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 inputs an ophthalmic image as an input image to the mathematical model, thereby obtaining a high-quality image by improving the quality of the input image. The ophthalmic image processing device 21 also obtains similarity information regarding the similarity between the input image and the high-quality image, and displays it on a confirmation screen 30 (see FIGS. 4 and 6) described later. The ophthalmic image capturing device 11 captures ophthalmic images, which are images of the tissues of the subject's eye.

[0041] As an example, a personal computer (hereinafter referred to as a "PC") is used as the mathematical model construction device 1 of this embodiment. As will be described in detail later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using ophthalmic images (hereinafter referred to as "training ophthalmic images") acquired from the ophthalmic image capturing device 11 and images with improved image quality of the training ophthalmic images (e.g., arithmetic average images, etc.). However, the device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic image capturing device 11 may function as the mathematical model construction device 1. Furthermore, control units of multiple devices (e.g., the CPU of the PC and the CPU 13 of the ophthalmic image capturing device 11) may cooperate to construct a mathematical model.

[0042] Furthermore, the ophthalmic image processing device 21 of this embodiment has a configuration for capturing ophthalmic images similar to those captured by the ophthalmic image capturing device 11. That is, in this embodiment, the ophthalmic image capturing device itself functions as the ophthalmic image processing device 21 that processes ophthalmic images. Therefore, the ophthalmic image processing device 21 of this embodiment can process the captured ophthalmic images while capturing them. This allows the user to determine the quality of the image (at least one of the input image and the high-quality image) immediately after the ophthalmic image is captured, making it easier to efficiently take measures such as recapturing the ophthalmic image. However, devices that can function as ophthalmic image processing devices are not limited to ophthalmic image capturing devices. For example, a PC may be used as the ophthalmic image processing device. Furthermore, control units of multiple devices may work together to process ophthalmic images.

[0043] In addition, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than a CPU may be used for at least some of the various devices. For example, a GPU may be used as a controller to speed up processing.

[0044] The mathematical model construction device 1 will now be described. The mathematical model construction device 1 is installed, for example, at a manufacturer that provides an ophthalmic image processing device 21 or an ophthalmic image processing program to users. The mathematical model construction device 1 includes a control unit 2 that performs various control processes and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller responsible for control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing the mathematical model construction process described below. The communication I / F 5 also connects the mathematical model construction device 1 to other devices (for example, an ophthalmic image capturing device 11 and an ophthalmic image processing device 21, etc.).

[0045] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the mathematical model construction device 1. The operation unit 7 can be, for example, at least one of a keyboard, a mouse, a touch panel, etc. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 7. The display device 8 displays various images. The display device 8 can be any of a variety of devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.). Note that, in this disclosure, "image" includes both still images and moving images.

[0046] The mathematical model construction device 1 can acquire data of ophthalmic images (hereinafter, sometimes simply referred to as "ophthalmic images") from the ophthalmic image capturing device 11. The mathematical model construction device 1 may acquire the data of ophthalmic images from the ophthalmic image capturing device 11 by, for example, at least one of wired communication, wireless communication, a removable storage medium (e.g., a USB memory), etc.

[0047] The ophthalmic image processing device (ophthalmic image capturing device) 21 will now be described. The ophthalmic image processing device 21 is installed, for example, in a facility (e.g., a hospital or health checkup facility) where a diagnosis or examination is performed on a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller responsible for control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see FIG. 3 ), which will be described later. The ophthalmic image processing program includes a program for realizing the mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (e.g., the mathematical model construction device 1, etc.). Furthermore, the ophthalmic image capturing device 21 of this embodiment includes an ophthalmic image capturing unit 26 for capturing ophthalmic images similar to those captured by the ophthalmic image capturing device 11. The ophthalmic image capturing unit 26 may have the same configuration as the ophthalmic image capturing unit 16 (details of which will be described later) included in the ophthalmic image capturing device 11.

[0048] The ophthalmologic image processing device 21 is connected to an operation unit 27 and a display device 28. As with the operation unit 7 and display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0049] The ophthalmic image capturing device 11 will be described. As an example, in this embodiment, a case will be described in which an ophthalmic image capturing device 11 that provides ophthalmic images to the mathematical model construction device 1 and an ophthalmic image capturing device that functions as an ophthalmic image processing device 21 are used. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from multiple ophthalmic image capturing devices. Furthermore, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a single common ophthalmic image capturing device.

[0050] In this embodiment, an OCT device is exemplified as the ophthalmic imaging device 11. However, an ophthalmic imaging device other than an OCT device (for example, a scanning laser ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

[0051] The ophthalmic image capturing device 11 includes a control unit 12 that performs various control processes, and an ophthalmic image capturing section 16. The control unit 12 includes a CPU 13 that is a controller that manages control, and a storage device 14 that can store programs, data, etc. When the ophthalmic image capturing device 11 executes at least a part of the ophthalmic image processing (see FIG. 3 ), which will be described later, it goes without saying that at least a part of the ophthalmic image processing program for executing the ophthalmic image processing is stored in the storage device 14.

[0052] The ophthalmologic image capturing unit 16 includes various components necessary for capturing ophthalmologic images of the subject's eye. The ophthalmologic image capturing unit 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit that scans the measurement light, an optical system that irradiates the subject's eye with the measurement light, and a light receiving element that receives a composite light of the light reflected by the tissue of the subject's eye and the reference light.

[0053] The ophthalmologic image capturing device 11 and the ophthalmologic image processing device 21 can capture two-dimensional tomographic images and three-dimensional tomographic images of tissue of the subject's eye (for example, fundus tissue in this embodiment). In particular, the CPU 13 scans OCT light (measurement light) along a scan line to capture two-dimensional tomographic images of a cross section intersecting the scan line. The CPU 13 can also capture three-dimensional tomographic images of the tissue by two-dimensionally scanning the OCT light.

[0054] Furthermore, the ophthalmologic image capturing device 11 and the ophthalmologic image processing device 21 can capture motion contrast images of tissues of the subject's eye (for example, fundus tissue in this embodiment). A motion contrast image is an image acquired by processing multiple OCT signals acquired at different times for the same position. The motion contrast image displays information about the movement of the subject. Therefore, the motion contrast image of this embodiment allows the blood vessels of the fundus of the subject's eye to be grasped non-invasively. This embodiment illustrates a case in which a two-dimensional motion contrast image of biological tissue viewed from the front along the optical axis of the OCT measurement light is captured and processed. The two-dimensional motion contrast image may be a two-dimensional enface image of the captured area of ​​a three-dimensional image viewed from the front. The data of the Enface image may be, for example, integrated image data in which brightness values ​​are integrated in the depth direction (Z direction) at each position in the direction (XY direction) intersecting the optical axis of the OCT measurement light, integrated values ​​of spectral data at each position in the XY direction, brightness data at each position in the XY direction in a certain depth direction, brightness data at each position in the XY direction in any layer of biological tissue (e.g., the retinal surface layer, etc.), etc. The ophthalmic image capturing device 11 and the ophthalmic image processing device 21 can also perform tracking processing to make the scanning position of the OCT light follow the movement of the subject's eye.

[0055] (Mathematical model construction process) 2, a description will be given of a mathematical model construction process executed by the mathematical model construction device 1. The mathematical model construction process is executed by the CPU 3 in accordance with a mathematical model construction program stored in the storage device 4.

[0056] In the mathematical model construction process, a mathematical model is trained using a training dataset to construct a mathematical model that outputs a high-quality image obtained by improving (converting) the quality of an input image. The training dataset includes input data (input training data) and output data (output training data). The mathematical model is capable of converting various ophthalmic images into high-quality images. The type of training dataset used to train the mathematical model is determined depending on the type of ophthalmic image whose quality is to be converted by the mathematical model. Below, we will explain a case in which a two-dimensional motion contrast image of a fundus taken from the front is input to the mathematical model as an input image, and a motion contrast image (high-quality image) obtained by improving the quality of the input image is output from the mathematical model as a converted image.

[0057] FIG. 2 shows an example of input training data and output training data when high-quality ophthalmic images (motion contrast images) are output to a mathematical model. In the example shown in FIG. 2, a CPU 3 acquires a set 60 of multiple ophthalmic images 600A-600X captured of the same tissue region. The CPU 3 uses a portion of the multiple ophthalmic images 600A-600X in the set 60 (a number smaller than the number used for averaging the output training data described below) as input training data. The CPU 3 also acquires an average image 61 of the multiple ophthalmic images 600A-600X in the set 60 as output training data. When a mathematical model is trained using the input training data and output training data shown in FIG. 2, the ophthalmic images are input as input images to the trained mathematical model, and high-quality ophthalmic images are output by the mathematical model. Note that the method for generating high-quality ophthalmic images used as output training data can be changed. For example, the image quality of the output training data may be improved by processing other than averaging.

[0058] The mathematical model construction process will now be described. The CPU 3 acquires at least a portion of the ophthalmologic images captured by the ophthalmologic image capturing device 11 as input training data. Next, the CPU 3 acquires output training data corresponding to the input training data. An example of the correspondence between the input training data and the output training data has been described above.

[0059] Next, the CPU 3 trains a mathematical model using the training dataset using a machine learning algorithm, such as a neural network, a random forest, boosting, or a support vector machine (SVM).

[0060] Neural networks are a method of imitating the behavior of biological neuronal networks. Examples of neural networks include feedforward neural networks, RBF networks (radial basis functions), spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0061] Random forest is a method to generate a large number of decision trees by learning from randomly sampled training data. When using random forest, the branches of multiple decision trees that have been trained as classifiers are traced, and the results obtained from each decision tree are averaged (or voted by majority vote).

[0062] Boosting is a technique for generating a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple weak classifiers.

[0063] SVM is a method for constructing a two-class pattern classifier using linear input elements. SVM learns the parameters of the linear input elements based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

[0064] A mathematical model refers to, for example, a data structure for predicting the relationship between input training data and output training data. A mathematical model is constructed by training using a training dataset. As described above, a training dataset is a set of input training data and output training data. For example, the correlation data (e.g., weights) between each input and output is updated through training.

[0065] In this embodiment, a multi-layer neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also referred to as units) are arranged in each layer. In detail, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layer neural network, is used. However, other machine learning algorithms may also be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be adopted as the machine learning algorithm.

[0066] The above process is repeated until the construction of the mathematical model is completed. When the construction of the mathematical model is completed, a program and data for realizing the constructed mathematical model are installed in the ophthalmologic image processing device 21.

[0067] (Ophthalmological Image Processing) 3 to 7, an example of ophthalmic image processing executed by the ophthalmic image processing device 21 will be described. The ophthalmic image processing illustrated in FIG. 3 is executed by the CPU 23 in accordance with an ophthalmic image processing program stored in the storage device 24.

[0068] As shown in FIG. 3, the CPU 23 acquires an ophthalmic image of the tissue of the subject's eye captured by an ophthalmic image capturing device (S1). In S11 of this embodiment, a motion contrast image of the fundus tissue of the subject's eye (input image 41 shown in FIGS. 4 to 6) is acquired. As described above, in this embodiment, the ophthalmic image processing device 21 also serves as an OCT device, which is a type of ophthalmic image capturing device. Therefore, in S1, the CPU 23 captures an ophthalmic image by controlling the ophthalmic image capturing unit 26, and acquires data of the captured ophthalmic image.

[0069] The CPU 23 displays a confirmation screen 30 (see FIGS. 4 and 6 ) on the display device 28 to allow the user to confirm the quality of the captured ophthalmic image (S2). The confirmation screen 30 illustrated in FIGS. 4 and 6 displays a related information display section 31, a superimposition selection section 33, a re-capture button 35, a delete button 36, and a save button 37. The related information display section 31 displays various related information about the ophthalmic images (an input image 41, a high-quality image 42, and a superimposed image 45, which will be described later) displayed on the confirmation screen 30. As an example, the related information display section 31 of this embodiment displays information about the date and time the ophthalmic image was captured, the type of ophthalmic imaging device that captured the ophthalmic image, the name of the examiner who performed the imaging using the ophthalmic imaging device, the ID of the subject whose ophthalmic image was captured, and the name of the subject whose ophthalmic image was captured. The superimposition selection section 33 is operated by the user to select a display mode for the superimposed image 45 (details of which will be described later). The delete button 36 is operated by the user when inputting an instruction to delete the ophthalmologic image data displayed on the confirmation screen 30 without saving it. The save button 37 is operated by the user when inputting an instruction to save the ophthalmologic image data displayed on the confirmation screen 30.

[0070] The CPU 23 inputs the ophthalmic image acquired in S1 as an input image 41 into a mathematical model trained by a machine learning algorithm, thereby converting the image quality of the input image 41 (in this embodiment, improving the image quality of the input image) to acquire a high-quality image 42 (S2). As shown in FIGS. 4 to 6, when the image quality conversion of the input image 41 is properly performed using the mathematical model, the high-quality image 42 acquired in S2 has a higher image quality than the input image 41. The input image 41 input to the mathematical model is generated without averaging or by averaging a small number of ophthalmic images. Therefore, it is easy to reduce the capture time of the input image 41. In contrast, when an image of a high quality equivalent to the high-quality image 42 shown in FIGS. 4 to 6 is captured using an ophthalmic imaging device, it is necessary to capture multiple images of the same part of the body and perform averaging, making it difficult to reduce the capture time. In this embodiment, the high-quality image 42 is acquired by inputting an input image 41 that can be captured in a short time into the mathematical model. Therefore, a high-quality image 42 is acquired while preventing the imaging time from becoming long.

[0071] Next, the CPU 23 acquires similarity information regarding the similarity at the same position between the input image 41 input to the mathematical model in S2 and the high-quality image 42 output from the mathematical model in S2 (S5). As an example, in this embodiment, the CPU 23 generates and acquires a similarity image 43 (see FIG. 5) that visualizes the similarity at the same position between the input image 41 input to the mathematical model in S2 and the high-quality image 42 output from the mathematical model in S2 (S5). As illustrated in FIG. 5, in areas where the accuracy of image quality conversion (image quality improvement) by the mathematical model is high, the similarity between the input image and the high-quality image indicated by the similarity image 43 tends to be high. On the other hand, in areas where the accuracy of image quality conversion by the mathematical model is low, the similarity between the input image and the high-quality image indicated by the similarity image 43 tends to be low. Therefore, the similarity image 43 appropriately reflects the distribution of accuracy of image quality conversion by the mathematical model.

[0072] As an example, in this embodiment, the CPU 23 normalizes the similarity of image information (pixel values) between regions or pixels at the same position in the input image 41 and the high-quality image 42, and represents each pixel of the similarity image 43 based on the normalized value. However, the method of representing the similarity image 43 can be changed. For example, the CPU 23 may obtain the difference in image information between regions or pixels at the same position in the input image 41 and the high-quality image 42. The smaller the difference in image information at the same position, the higher the similarity. Therefore, the CPU 23 can represent each pixel of the similarity image 43 based on the difference in image information. Furthermore, the CPU 23 generates the similarity image 43 after performing a Fourier transform (e.g., a two-dimensional Fourier transform) on each of the input image 41 and the high-quality image 42. As a result, the presence or absence of periodic artifacts can be more appropriately evaluated.

[0073] Next, the CPU 23 displays a superimposed image 45 (see FIGS. 4 and 6 ) on the confirmation screen 30 (S6), in which the similarity image 43 generated in S5 is superimposed (for example, superimposed in a registered state) on at least one of the input image 41 and the high-quality image 42. By checking the superimposed image 45, the user can properly grasp the distribution of accuracy of the image conversion by the mathematical model on the ophthalmic image (input image 41 or high-quality image 42) on which the similarity image 43 is superimposed. Furthermore, in this embodiment, the superimposed image 45 is displayed on the confirmation screen 30 for allowing the user to confirm the quality of the ophthalmic image (input image 41 and high-quality image 42 in this embodiment). Therefore, the user can grasp the distribution of accuracy of the image quality conversion by the mathematical model when confirming the quality of the input image 41 after capturing it. Therefore, the user can properly determine the quality of the captured or acquired ophthalmic image and then efficiently and appropriately take measures, such as re-capturing the ophthalmic image.

[0074] However, the superimposed image 45 may be displayed on a screen other than the confirmation screen 30 (for example, a screen displayed during medical examination after the ophthalmological image is captured). Even in this case, by checking the superimposed image 45, the distribution of accuracy of image conversion by the mathematical model can be properly grasped on the ophthalmological image on which the similarity image 43 is superimposed.

[0075] As described above, the ophthalmologic images captured, acquired, and processed in this embodiment are motion contrast images. Motion contrast images capture information about the movement of the subject. Therefore, the motion contrast images of this embodiment enable non-invasive identification of blood vessels in the fundus. However, if a motion contrast image is captured for the purpose of identifying blood vessels, and biological tissues other than blood vessels move during imaging, the effects of the biological tissue movement will also be reflected in the motion contrast image. In this case, the quality of at least one of the input image 41 and the high-quality image 42 will deteriorate. Furthermore, various image-related parameters also affect the quality of the input image 41 and the high-quality image 42. However, even when a user views the input image 41 and the high-quality image 42, it is often difficult to determine whether the quality has deteriorated. In contrast, when a high-quality image 42 is obtained by inputting a motion contrast image as the input image 41 into a mathematical model, the accuracy of image quality conversion by the mathematical model is significantly affected by the quality of the motion contrast image input to the mathematical model. Therefore, a high-quality image 42 is obtained using a motion contrast image as an input image 41, and a similarity image 43 between the input image 41 and the high-quality image 42 is superimposed on at least one of the input image 41 and the high-quality image 42, making it easier to properly understand the distribution of quality of the ophthalmic image on which the similarity image 43 is superimposed.

[0076] In S5, the CPU 23 increases the transparency of the pixels constituting the similarity image 43 as the similarity between the pixel values ​​of the input image 41 and the high-quality image 42 decreases, and displays the similarity image 43 superimposed on the ophthalmological image (input image 41 or high-quality image 42). As a result, in areas where the similarity between the input image 41 and the high-quality image 42 is low, the user can easily grasp the state of the ophthalmological image on which the similarity image 43 is superimposed. Therefore, the user can easily grasp the state of areas where the accuracy of image quality conversion was low in the ophthalmological image on which the similarity image 43 is superimposed, for example.

[0077] Next, the CPU 23 determines whether or not an instruction to select the display mode of the superimposed image 45 has been input by the user (S8). If an instruction has not been input (S8: NO), the process proceeds directly to S10. If an instruction has been input (S8: YES), the CPU 23 switches the display mode of the superimposed image 45 displayed on the confirmation screen 30 in accordance with the instruction input by the user (S9). Thereafter, the process proceeds to S10.

[0078] As shown in FIGS. 4 and 6, the confirmation screen 30 of this embodiment displays a superimposition selection section 33 that allows the user to select a display mode of the superimposed image 45. In the example shown in FIGS. 4 and 6, when the selection box "On input image" is selected (S8: YES), the CPU 23 displays a superimposed image 45 in which the similarity image 43 is superimposed on the input image 41 (S9). When the selection box "On high quality image" is selected (S8: YES), the CPU 23 displays a superimposed image 45 in which the similarity image 43 is superimposed on the high quality image 42 (S9). When the selection box "Not superimposed" is selected (S8: YES), the CPU 23 switches between superimposing and not superimposing the similarity image 43 on the superimposed image 45 (S9). Thus, the user can properly grasp both the ophthalmological image with the similarity image 43 superimposed and the ophthalmological image without the similarity image 43 superimposed.

[0079] As described in S8 and S9, in this embodiment, the CPU 23 can display a superimposed image 45, in which the similarity image 43 is superimposed on the high-quality image 42, on the confirmation screen 30. The accuracy of image quality conversion using a mathematical model may be reduced by various influences (e.g., the influence of tissue movement during imaging, the influence of imaging conditions, the influence of image parameters, etc.). The similarity image 43, which visualizes the similarity between the input image 41 and the high-quality image 42, tends to clearly show the distribution of areas where the accuracy of image quality conversion using the mathematical model is low and areas where it is high. Therefore, by superimposing the similarity image 43 on the high-quality image 42, the user can appropriately grasp the presence or absence and distribution of areas in the high-quality image 42 where the accuracy of image quality conversion is low.

[0080] Furthermore, in this embodiment, the CPU 23 can display a superimposed image 45, in which the similarity image 43 is superimposed on the input image 41, on the confirmation screen 30. If at least a portion of the input image 41 contains an area of ​​poor quality (hereinafter referred to as a "bad area"), when the image quality of the input image 41 is improved using a mathematical model (i.e., a high-quality image 42 is obtained), the accuracy of the image quality conversion in the bad area is likely to be lower than the accuracy of the image quality conversion in an area of ​​good quality (hereinafter referred to as a "good area"). Therefore, in an area where the similarity indicated by the similarity image 43 is low (i.e., an area where the accuracy of the image quality conversion is low), the quality of the input image 41 input to the mathematical model is likely to be poor. Therefore, by superimposing the similarity image 43 on the input image 41, the user can appropriately grasp the presence or absence and distribution of bad areas in the input image 41.

[0081] Next, the CPU 23 determines whether the save button 37 or the delete button 36 (see FIGS. 4 and 6) has been operated by the user (S10). When the save button 37 or the delete button 36 is operated (S10: YES), the CPU 23 executes a process of saving the input image 41 and the high-quality image 42 displayed on the confirmation screen 30 to the storage device 24 or a process of deleting them, in accordance with the instruction input by the user (i.e., depending on whether the save button 37 or the delete button 36 has been operated), and the process ends (S11). As described above, when the user confirms from the superimposed image 45 displayed on the confirmation screen 30 that the image quality of the input image 41 and the high-quality image 42 is appropriate, the user can easily save the input image 41 and the high-quality image 42 in the storage device 24.

[0082] If neither the save button 37 nor the delete button 36 has been operated (S10: NO), the CPU 23 determines whether or not an instruction to designate a re-capture area has been input by the user (S12). The re-capture area is an area within the image area (in this embodiment, an image area common to all of the input image 41, the high-quality image, and the superimposed image 45) where an image is to be re-captured. If an instruction to designate a re-capture area has not been input (S12: NO), the process proceeds directly to S15. If an instruction to designate a re-capture area has been input (S12: YES), the CPU 23 sets a re-capture area in accordance with the instruction input by the user (S13), and the process proceeds to S15.

[0083] The processing of S12 and S13 in this embodiment will be described in detail with reference to Fig. 6. As shown in Fig. 6, the CPU 23 can accept input of an instruction from the user to specify the re-capture area 50 on the superimposed image 45 displayed on the confirmation screen 30. In this embodiment, the user can specify the re-capture area 50 on the superimposed image 45 by operating the operation unit 27 to specify at least one of the position, shape, and size of the frame of the re-capture area 50 displayed on the superimposed image 45. Therefore, the user can easily and appropriately specify the re-capture area 50 on the superimposed image 45 after understanding the distribution of accuracy of image quality conversion using a mathematical model from the superimposed image 45.

[0084] The input image 41, the high-quality image 42, and the superimposed image 45 displayed on the confirmation screen 30 share the same image area. In this embodiment, the CPU 23 displays a frame of the re-captured area 50, which has the same position, shape, and size, on each of the input image 41, the high-quality image 42, and the superimposed image 45. This allows the user to grasp the re-captured area 50 not only on the superimposed image 45, but also on the input image 41 and the high-quality image 42. When the CPU 23 changes at least one of the position, shape, and size of the frame of the re-captured area 50 displayed on the superimposed image 45 in response to a user instruction, the CPU 23 also changes the frame of the re-captured area 50 on the input image 41 and the high-quality image 42 accordingly. In this embodiment, the user can also specify the positions of the frame of the re-captured area 50 on the input image 41 and the high-quality image 42. In either case, the CPU 23 changes the positions of the frames of the multiple re-photographed regions 50 in conjunction with each other in response to an instruction from the user.

[0085] Next, the CPU 23 determines whether or not the user has input an instruction to re-capture an ophthalmic image (S15). If the instruction to re-capture has not been input (S15: NO), the process returns to S8, and the processes of S8 to S15 are repeated. If the user operates the re-capture button 35 (see FIGS. 4 and 6) on the confirmation screen 30 to input an instruction to re-capture (S15: YES), the CPU 23 outputs an instruction to re-capture an ophthalmic image for a position on the tissue corresponding to the re-capture area 50 set in S13 (i.e., specified in S12) within the image area of ​​the superimposed image 45, and acquires the re-captured ophthalmic image (S16). As a result, ophthalmic images of areas where the accuracy of image quality conversion was low are re-captured in an easy and appropriate procedure.

[0086] In S15 of this embodiment, the CPU 23 accepts input of an instruction to re-capture the ophthalmic image while displaying the superimposed image 45 on the confirmation screen 30. When the instruction to re-capture is input, the CPU 15 re-captures the ophthalmic image (S16). Therefore, the user can properly grasp the distribution of accuracy of image quality conversion by the mathematical model on the ophthalmic image on which the similarity image 43 is superimposed, and if the accuracy of the image quality conversion is low, can immediately re-capture the ophthalmic image. Therefore, the process of re-capturing the image can be efficiently and appropriately executed.

[0087] Furthermore, in S16 of the present embodiment, when a command to perform re-imaging is input by the user, the CPU 23 can re-imaging an ophthalmological image for a tissue range corresponding to a part of the re-imaging region 50 in the image region of the superimposed image 45. Therefore, for example, it is possible to re-imaging a part of the image region of the superimposed image 45 having a low similarity indicated by the similarity image 43 as the re-imaging region 50. Therefore, the time required for re-imaging can be shortened compared to when an ophthalmological image is re-imaging for a tissue range corresponding to the entire image region of the superimposed image 45.

[0088] 7, the CPU 23 replaces an image corresponding to the re-captured area 50 set in S13 in the original input image 41O, which is the input image before the re-capture in S16, with the re-captured image 41R acquired in S16, thereby generating a new input image 41N (S17). That is, the re-captured image 41R, which was re-captured in a short time in S16, is combined with an appropriate area in the original input image 41O to generate a new input image 41N. Thus, an appropriate input image is efficiently acquired. The process returns to S4, and the new input image 41N generated in S17 is input into the mathematical model, thereby acquiring a new high-quality image 42. Thereafter, the processes from S5 onwards are repeated.

[0089] The CPU 23 can also re-photograph an ophthalmologic image for a tissue range corresponding to the entire image area of ​​the superimposed image 45. In this embodiment, when a re-photograph execution instruction is input in S15 in a state where the re-photograph area 50 has not been set in S12 and S13, the CPU 23 re-photographs an ophthalmologic image for a tissue range corresponding to the entire image area. In this case, the CPU 23 replaces the entire image area of ​​the ophthalmologic image before re-photographing (the original input image 41O shown in FIG. 7) with the re-photographed image 41R. The same applies when the re-photograph area 50 is set to the entire image area of ​​the superimposed image 45 in S12 and S13.

[0090] The techniques disclosed in the above embodiments are merely examples. Therefore, the techniques exemplified in the above embodiments can be modified. First, it is possible to implement only some of the techniques exemplified in the above embodiments. For example, it is possible to omit the process for re-capturing the ophthalmologic image of a part of the re-captured area 50 in the image area of ​​the superimposed image 45.

[0091] 4 and 6, the input image 41, the high-quality image 42, and the superimposed image 45 are displayed. However, the display of at least one of the input image 41 and the high-quality image 42 may be omitted. For example, when the superimposed image 45 in which the similarity image 43 is superimposed on the input image 41 is displayed, the display of the input image 41 alone may be omitted. Furthermore, when the superimposed image 45 in which the similarity image 43 is superimposed on the high-quality image 42 is displayed, the display of the high-quality image 42 alone may be omitted.

[0092] The process of acquiring an ophthalmologic image in S1 of FIG. 3 is an example of an "image acquiring step." The process of displaying the confirmation screen 30 in S2 is an example of a "confirmation screen display step." The process of acquiring a high-quality image 42 in S4 is an example of a "high-quality image acquiring step." The process of generating and acquiring a similarity image 43 (an example of similarity information) in S5 is an example of a "similarity information acquiring step." The process of displaying a superimposed image 45 in S6 is an example of a "similarity information display step." The process of accepting input of an instruction to perform re-imaging in S15 is an example of a "re-imaging instruction accepting step." The process of performing re-imaging in S16 is an example of a "re-imaging step." The process of replacing the re-imaging image in S17 is an example of a "replacement step." The process of accepting input of an instruction to specify a re-imaging area in S12 is an example of an "area accepting step." [Explanation of symbols]

[0093] 11 Ophthalmic imaging device 21 Ophthalmological image processing device 23 CPU 24 Storage device 28 Display device 30 Confirmation screen 41 input images 42 high-quality images 43 Similarity Images 45 Superimposed Images 50 Re-photographed area

Claims

1. An ophthalmic image processing program executed by an ophthalmic image processing device that processes an ophthalmic image, which is an image of tissue of a subject's eye, The ophthalmologic image processing program is executed by a control unit of the ophthalmologic image processing device, an image acquisition step of acquiring an ophthalmologic image captured by an ophthalmologic image capturing device; a confirmation screen display step of displaying a confirmation screen on a display unit to allow a user to confirm whether the ophthalmologic image captured by the ophthalmologic image capturing device is good or bad; a high-quality image acquisition step of acquiring a high-quality image by improving the quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm; a similarity information acquisition step of acquiring similarity information which is information regarding the similarity between the input image and the high-quality image at the same position; a similarity information display step of displaying the similarity information on the confirmation screen; 10. An ophthalmologic image processing program, wherein the program is executed by the ophthalmologic image processing device.

2. 2. The ophthalmologic image processing program according to claim 1, An ophthalmic image processing program characterized in that the ophthalmic image is a motion contrast image obtained by processing at least two OCT signals obtained at different times from the same position by an OCT device, which is the ophthalmic image capturing device.

3. 3. The ophthalmologic image processing program according to claim 1, In the similarity information acquisition step, a similarity image is generated as the similarity information by visualizing the similarity at the same position between the input image and the high-quality image, and the similarity information is acquired; The ophthalmologic image processing program is characterized in that, in the similarity information display step, the similarity image is displayed on the confirmation screen.

4. 4. The ophthalmologic image processing program according to claim 3, An ophthalmologic image processing program characterized in that, in the similarity information display step, a superimposed image in which the similarity image is superimposed on at least one of the input image and the high-quality image is displayed on the confirmation screen.

5. 5. The ophthalmologic image processing program according to claim 4, An ophthalmological image processing program characterized in that, in the similarity information display step, the lower the similarity between the pixel values ​​of the input image and the high-resolution image, the higher the transparency of the pixels that make up the similarity image, and the similarity image is superimposed on at least one of the input image and the high-resolution image.

6. 6. An ophthalmologic image processing program according to claim 4, a re-imaging instruction receiving step of receiving an input of an instruction to re-imaging an ophthalmologic image while the superimposed image is displayed on the confirmation screen; a re-photographing step of performing re-photographing of an ophthalmologic image by the ophthalmologic image photographing device when a re-photographing instruction is input by a user in the re-photographing instruction receiving step; an ophthalmologic image processing program, wherein the program is further executed by the ophthalmologic image processing device.

7. 7. An ophthalmologic image processing program according to claim 6, an ophthalmologic image processing program characterized in that, in the re-photographing step, when an instruction to perform re-photographing is input by a user, re-photographing of an ophthalmologic image is performed for a tissue range corresponding to a part of the re-photographed area of ​​the image area of ​​the superimposed image.

8. 8. An ophthalmologic image processing program according to claim 7, a replacement step of, when re-imaging of an ophthalmic image of a tissue range corresponding to the part of the re-imaging region is performed in the re-imaging step, replacing an image within the re-imaging region in the ophthalmic image before re-imaging is performed with the re-imaging region; an ophthalmologic image processing program, wherein the program is further executed by the ophthalmologic image processing device.

9. 9. An ophthalmologic image processing program according to claim 6, an area receiving step of receiving an input of an instruction to designate a re-photographing area for re-photographing an ophthalmologic image on the superimposed image displayed on the confirmation screen is further executed; an ophthalmologic image processing program characterized in that, in the re-photographing step, re-photographing of an ophthalmologic image is performed for a tissue range corresponding to the re-photographed area specified in the area receiving step among the image areas of the superimposed image.

10. 10. An ophthalmologic image processing program according to claim 4, The ophthalmologic image processing program is characterized in that, in the similarity information display step, a superimposed image in which the similarity image is superimposed on the high-quality image is displayed on the confirmation screen.

11. 11. An ophthalmologic image processing program according to claim 4, The ophthalmologic image processing program, wherein the similarity information display step displays a superimposed image obtained by superimposing the similarity image on the input image on the confirmation screen.

12. An ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of a subject's eye, The control unit of the ophthalmologic image processing device an image acquisition step of acquiring an ophthalmologic image captured by an ophthalmologic image capturing device; a confirmation screen display step of displaying a confirmation screen on a display unit to allow a user to confirm whether the ophthalmologic image captured by the ophthalmologic image capturing device is good or bad; a high-quality image acquisition step of acquiring a high-quality image by improving the quality of the input image by inputting the ophthalmic image acquired in the image acquisition step as an input image into a mathematical model trained by a machine learning algorithm; a similarity information acquisition step of acquiring similarity information which is information regarding the similarity between the input image and the high-quality image at the same position; a similarity information display step of displaying the similarity information on the confirmation screen; An ophthalmological image processing device characterized by executing the above.

Images