Ophthalmic image processing program and ophthalmic image processing device

Abstract

To provide an ophthalmologic image processing program and an ophthalmologic image processing device capable of appropriately presenting medical information desired by a user to the user.SOLUTION: A control unit of an ophthalmologic image processing device acquires an ophthalmologic image captured by an OCT device (S11). The control unit acquires a high quality image obtained by improving the quality of a base image as medical information by inputting the base image to a mathematical model trained by a machine learning algorithm with a partial image, which is a part of the acquired ophthalmologic image, or an image generated on the basis of the partial image as the base image. When an instruction to select the partial image is received, the control unit causes medical information acquired on the selected partial image to be displayed in a display unit (S28). The control unit executes preliminary acquisition processing for preliminarily acquiring and storing medical information on a partial image yet to be selected of the ophthalmologic image (S22).SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an ophthalmic image processing program and an ophthalmic image processing apparatus used for processing an ophthalmic image of a subject's eye. [Background technology]

[0002] A technique for acquiring various medical information using a mathematical model trained by a machine learning algorithm has been proposed. For example, in an ophthalmologic apparatus described in Patent Document 1, eye shape parameters are inputted into a mathematical model trained by a machine learning algorithm to acquire IOL-related information (e.g., predicted postoperative anterior chamber depth) of a subject's eye. Based on the acquired IOL-related information, an IOL power is calculated.

[0003] It is also possible to obtain a high-quality image from a low-quality image using a mathematical model trained by a machine learning algorithm, where the low-quality image is input to the mathematical model and a high-quality image is output. [Prior art documents] [Non-patent literature]

[0004] [Patent Document 1] JP 2018-51223 A Summary of the Invention [Problem to be solved by the invention]

[0005] A user may want to select a portion of an image (hereinafter referred to as a "partial image") of interest from the entire captured image, and check medical information (e.g., a high-quality image, etc.) related to the selected partial image. In this case, the image processing device may acquire medical information related to the selected partial image using a mathematical model each time a partial image is selected by the user, and present the acquired medical information to the user. However, since the process of acquiring medical information using a mathematical model takes time, it is difficult to reduce the user's work time with a method of acquiring and presenting medical information each time a partial image is selected. Therefore, it is desirable to be able to appropriately present the medical information desired by the user to the user.

[0006] A typical object of the present disclosure is to provide an ophthalmologic image processing program and an ophthalmologic image processing device that are capable of appropriately presenting medical information desired by the user to the user. [Means for solving the problem]

[0007] An ophthalmic image processing program provided by an exemplary embodiment of the present disclosure is an ophthalmic image processing program executed by an ophthalmic image processing device that processes ophthalmic images, which are images of tissues of a subject's eye, and the ophthalmic image processing program is executed by a control unit of the ophthalmic image processing device, thereby performing an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT device, a medical information acquisition step of generating an enface image when the partial image is viewed from a direction along the optical axis of the measurement light of the OCT device based on a partial image, which is a three-dimensional tomographic image of a partial area sandwiched between two boundary surfaces, of the three-dimensional tomographic image acquired in the image acquisition step, and inputting the enface image as a base image into a mathematical model trained by a machine learning algorithm to acquire a high-quality image in which the image quality of the base image is improved, and storing the high-quality image in a storage device, and a medical information acquisition step of acquiring a high-quality image in the three-dimensional tomographic image acquired in the image acquisition step. , which is a three-dimensional cross-sectional image of a part of the region between the two boundaries.The ophthalmologic image processing device executes an instruction receiving step of receiving an input of an instruction to select a partial image, and a medical information display step of, when the instruction to select a partial image is received, causing a display unit to display a high-quality image acquired for the partial image selected by the instruction, wherein in the medical information acquisition step, a pre-acquisition process is executed to acquire and store in advance a high-quality image for a partial image among the three-dimensional tomographic image that has not yet been selected by the instruction to select a partial image, before the input of the instruction to select the partial image is actually received in the instruction receiving step, and the pre-acquisition process is executed in parallel with the instruction receiving step of receiving an input of an instruction to select a partial image and the medical information display step of displaying on the display unit a high-quality image acquired for the partial image selected by the instruction to select a partial image.

[0008] An ophthalmic image processing device provided by an exemplary embodiment of the present disclosure is an ophthalmic image processing device that processes ophthalmic images that are images of tissues of a subject's eye, and a control unit of the ophthalmic image processing device includes an image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT device, a medical information acquisition step of generating an enface image when the partial image is viewed from a direction along the optical axis of the measurement light of the OCT device based on a partial image that is a three-dimensional tomographic image of a partial area sandwiched between two boundary surfaces out of the three-dimensional tomographic image acquired in the image acquisition step, and inputting the enface image as a base image into a mathematical model trained by a machine learning algorithm to acquire a high-quality image in which the image quality of the base image is improved, and storing the high-quality image in a storage device, and , which is a three-dimensional cross-sectional image of a part of the region between the two boundaries.The method includes an instruction receiving step of receiving an input of an instruction to select a partial image, and a medical information display step of, when the instruction to select a partial image is received, causing a display unit to display a high-quality image acquired for the partial image selected by the instruction. In the medical information acquisition step, the control unit executes a pre-acquisition process to acquire and store high-quality images for partial images of the three-dimensional tomographic image that have not yet been selected by the instruction to select a partial image, before the input of the instruction to select the partial image is actually received in the instruction receiving step. The pre-acquisition process is executed in parallel with the instruction receiving step of receiving an input of an instruction to select a partial image and the medical information display step of displaying a high-quality image acquired for the partial image selected by the instruction to select a partial image on the display unit.

[0009] According to the ophthalmologic image processing program and ophthalmologic image processing device according to the present disclosure, medical information desired by the user is appropriately presented to the user. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image photographing devices 11A and 11B. [Diagram 2] 13A and 13B are diagrams showing an example of input training data and output training data when a high-quality two-dimensional tomographic image is output to a mathematical model. [Diagram 3] 1 is a diagram showing an example of a three-dimensional tomographic image 50 and an enface image 55 generated based on a partial image included in the three-dimensional tomographic image 50. FIG. [Figure 4] 2 is a flowchart of a mathematical model construction process executed by the mathematical model construction device 1. [Diagram 5] 4 is a flowchart of ophthalmologic image processing executed by the ophthalmologic image processing device 21. [Figure 6] 1 is an example of a display screen 60 displayed on the display device 28 during execution of ophthalmologic image processing. [Figure 7] 1 is a diagram showing an example of a partial image 66 selected by a user, and a high-quality image 67 acquired from the partial image 66. FIG. [Figure 8] FIG. 13 is a diagram showing an example of a setting screen displayed on a display device when a user sets a pre-acquired pattern. [Figure 9] FIG. 11 is an explanatory diagram for explaining an example of a method for setting priorities for a plurality of partial images 66 with reference to a reference image BL. [Figure 10] FIG. 11 is an explanatory diagram for explaining an example of a method for setting priorities for a plurality of partial images 66 along the direction in which the partial images 66 selected by the user are switched. [Figure 11] 13 is a flowchart of a pre-acquisition process executed by the ophthalmologic image-processing device 21. [Figure 12] 13 is an example of a display screen 80 displayed on the display device 28 during execution of ophthalmologic image processing in the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Summary> The control unit of the ophthalmic image processing device exemplified in the present disclosure executes an image acquisition step, a medical information acquisition step, an instruction receiving step, and a medical information display step. In the image acquisition step, the control unit acquires an ophthalmic image of the tissue of the subject's eye photographed by the OCT device. In the medical information acquisition step, the control unit acquires a high-quality image in which the image quality of the base image is improved by inputting a partial image of the acquired ophthalmic image or an image generated based on the partial image as a base image and inputting the base image into a mathematical model trained by a machine learning algorithm, as medical information, and stores the high-quality image in the storage device. In the instruction receiving step, the control unit receives an input of an instruction to select a partial image in the acquired ophthalmic image. In the medical information display step, when an instruction to select a partial image is received, the control unit displays medical information acquired regarding the partial image selected by the instruction on the display unit. In the medical information acquisition step, the control unit executes a pre-acquisition process to acquire and store medical information regarding a partial image not yet selected by the instruction among the ophthalmic images.

[0012] In the ophthalmologic image processing device exemplified in the present disclosure, medical information (high-quality image) related to a partial image that has not yet been selected by the user is acquired and stored in advance by a pre-acquisition process. Therefore, when medical information related to an actually selected partial image has already been stored in the storage device, the control unit can cause the display unit to display the medical information desired by the user in a short time. Therefore, the user can appropriately check the desired medical information.

[0013] Various ophthalmic images can be used as the ophthalmic image processed by the ophthalmic image processing device. For example, the ophthalmic image acquired in the image acquiring step may be a three-dimensional tomographic image including a plurality of two-dimensional tomographic images. In this case, the partial image may be one or more of the plurality of two-dimensional tomographic images included in the three-dimensional tomographic image. The control unit may use the two-dimensional tomographic image as the partial image as a base image, and acquire an image obtained by improving the image quality of the base image as a high-quality image.

[0014] Furthermore, when the ophthalmologic image is a three-dimensional tomographic image, the partial image may be a part of a three-dimensional tomographic image included in the entire image area of ​​the three-dimensional tomographic image (for example, a three-dimensional tomographic image in an area sandwiched between two boundary surfaces). The control unit may use an enface image (OCT front image) obtained when the three-dimensional tomographic image, which is a partial image, is viewed from a direction (front direction) along the optical axis of the measurement light of the OCT device as the base image. The enface image may be, for example, an integrated image obtained by integrating pixel values ​​of the partial image in the depth direction (Z direction) at each position in the XY directions intersecting the optical axis. The enface image may also be generated by integrating the spectral data of the partial image at each position in the XY directions. The control unit may obtain a high-quality image by inputting the base image, which is the enface image, into a mathematical model. The three-dimensional tomographic image itself, which is a partial image, may be used as the base image. The base image may also be an OCT angio image. The OCT angio image may be a two-dimensional front image obtained by viewing the fundus from the front (i.e., the line of sight of the subject eye). The OCT angiogram may be, for example, a motion contrast image obtained by processing at least two OCT signals acquired at different times for the same position.

[0015] The ophthalmologic image acquired in the image acquiring step may be a moving image including a plurality of still images acquired by capturing the same tissue intermittently a plurality of times in a time series. In this case, the partial image may be one or more of the plurality of still images constituting the moving image. The control unit may use the still image that is the partial image as a base image.

[0016] The ophthalmologic image acquired in the image acquiring step may be a two-dimensional image that is a still image. In this case, the partial image may be an image of a part of an area included in the area of ​​the two-dimensional image. The control unit may use the image of the part of the area that is the partial image as the base image.

[0017] In the pre-acquisition process, the control unit may sequentially execute the process of acquiring medical information for each of the partial images in the ophthalmic image in descending order of priority set for the partial images. In this case, medical information is acquired sequentially in an appropriate order for each of the partial images.

[0018] In the pre-acquisition process, the control unit may set one or more of the multiple partial images in the ophthalmologic image as a reference image, and set a higher priority for each of the multiple partial images as the partial images are closer to the reference image. In this case, medical information is preferentially acquired from partial images closer to the reference image. Therefore, by appropriately setting the reference image, medical information is acquired sequentially in a more appropriate order. This makes it easier to further shorten the user's work time.

[0019] In addition, various "closeness" can be adopted for the "closeness" of the partial images to the reference image, which is the criterion for setting the priority order. For example, when the partial images are two-dimensional tomographic images constituting a three-dimensional tomographic image, the "closeness" of the distance to the two-dimensional tomographic image set as the reference image may be used as the criterion.

[0020] Furthermore, when a part of a three-dimensional tomographic image included in an image region of the three-dimensional tomographic image is used as a partial image, at least one of the "closeness" of the distance to the three-dimensional tomographic image used as the reference image, the "closeness" of the range, the "closeness" of the overlapping ratio, etc. may be used as a criterion. Furthermore, when a three-dimensional tomographic image in a region sandwiched between two boundary surfaces is used as a partial image, the "closeness" of the distance of the boundary surface of the partial image to the boundary surface of the reference image may be used as a criterion.

[0021] In addition, when still images constituting a moving image are used as partial images, the "closeness" of the shooting time to the still image used as the reference image may be used as a criterion. In addition, when an image of a part of a region in a two-dimensional image is used as a partial image, at least one of the "closeness" of the distance, the range, and the overlap rate to the reference image may be used as a criterion.

[0022] In the pre-acquisition process, the control unit may acquire a specific position of the tissue in the ophthalmologic image, and may use a partial image at the acquired specific position as a reference image. In this case, medical information is preferentially acquired from the specific position of the tissue. Therefore, the medical information at the specific position can be appropriately grasped by the user.

[0023] The method of the control unit to acquire the specific position may be appropriately selected. For example, the control unit may detect the specific position of the tissue by performing well-known image processing on the ophthalmic image. Alternatively, the control unit may acquire the specific position by inputting the acquired ophthalmic image into a mathematical model that inputs the ophthalmic image and outputs the specific position.

[0024] The type of the specific position acquired by the control unit can also be appropriately selected. For example, when the ophthalmologic image is a fundus image of the subject's eye, the specific position may be at least one of the fovea, the optic disc, a lesion, and a fundus blood vessel in the fundus. When the fundus image is a three-dimensional tomographic image of the fundus, the specific position may be the position of a specific layer in the fundus.

[0025] The specific position may be acquired by using various ophthalmic examination devices. As an example, the specific position may be a position identified by a perimeter, which is a type of ophthalmic examination device (for example, a position on the fundus with low sensitivity). In this case, the control unit may acquire the specific position based on an image of the examination result by the perimeter.

[0026] However, the reference image may be set regardless of the specific position. For example, the reference image may be a partial image including the center of the image area of ​​the ophthalmic image.

[0027] When an instruction to select the partial image is received, the control unit may set the partial image selected by the instruction as a reference image. When a user selects a partial image, the user often also wants to check medical information related to a partial image similar to the selected partial image. By setting the partial image selected by the user as the reference image, the control unit can acquire and store medical information about partial images similar to the selected partial image by a pre-acquisition process. As a result, the medical information of the partial image similar to the selected partial image is displayed on the display unit in a short time. Therefore, the user can more appropriately check the desired medical information.

[0028] In addition, when medical information on the partial image selected by the instruction has not yet been acquired, the control unit may acquire the medical information on the selected partial image as a top priority in the medical information acquisition step and display it on the display unit in the medical information display step. In addition, when medical information on the partial image selected by the instruction has been acquired in advance by a pre-acquisition process, the control unit may display the already acquired medical information on the display unit in the medical information display step. As a result, the medical information desired by the user is smoothly displayed on the display unit.

[0029] When a partial image to be selected is changed by a user's instruction, the control unit may set the priority order for the multiple partial images in order along the direction of the changed partial image relative to the partial image before the change. Users often switch partial images in order along the same direction. By setting the priority order for the multiple partial images in order along the direction in which the partial images are switched, the control unit can acquire and store in advance medical information of partial images that are likely to be selected later by the user. Thus, the user can more appropriately check the desired medical information.

[0030] When the control unit sets the partial image selected by the user as the reference image, the control unit may store the position of the reference image in the ophthalmic image in the storage device. When processing an ophthalmic image having the same subject as an ophthalmic image whose position of the reference image has been stored in the past, the control unit may set the partial image of the ophthalmic image at the previously stored position as the reference image. When ophthalmic images are taken at different times for the same tissue of the subject's eye to perform diagnosis, etc., the user often checks medical information at the same position in the ophthalmic image. By setting the partial image at the previously stored position as the reference image, the control unit can efficiently acquire medical information at a position that the user is likely to check in advance.

[0031] When a new reference image is set, the control unit may interrupt the pre-acquisition process that has been executed, and resume the pre-acquisition process according to the priority order based on the newly set reference image. In this case, medical information of a partial image close to the newly set reference image is acquired in a shorter time than when the pre-acquisition process based on the newly set reference image is executed after the pre-acquisition process that has been executed is completed. Therefore, the user can more appropriately check the desired medical information.

[0032] The control unit may resume the interrupted pre-acquisition process when the pre-acquisition process based on the newly set reference image is completed. Also, when the control unit is sequentially executing the pre-acquisition process for each of the multiple partial images according to the priority order, the control unit may omit the pre-acquisition process for the partial images for which medical information has already been acquired. In this case, the medical information for each of the multiple partial images is acquired more efficiently.

[0033] Among a plurality of partial images in an ophthalmologic image, a pattern of partial images for which medical information is acquired in advance may be provided in advance. The control unit may execute an acquisition process of medical information related to the partial images corresponding to the pattern in the advance acquisition process. In this case, the medical information related to the partial images corresponding to the pattern is acquired in advance. Therefore, the user can appropriately grasp the medical information corresponding to the pattern.

[0034] The specific form of the pattern can be appropriately selected. For example, when two-dimensional tomographic images constituting a three-dimensional tomographic image of the fundus are used as partial images, the pattern may be provided so that the arrangement of the partial images when the fundus is viewed from the front is a cross, radial, concentric, raster, or a combination of these.

[0035] In addition, when a three-dimensional tomographic image in a region between two boundary surfaces of a three-dimensional image of the fundus is used as a partial image, a pattern of partial images in which at least one of the positions of the two boundary surfaces is preset (for example, a pattern in which the layer between the two boundary surfaces is at least one of the retinal surface layer, the deep retina layer, the vitreous body, the outer retina layer, the ORCC, the choriocapillaris plate, the choroid, etc.) may be set. In particular, when a three-dimensional image in a region between two boundary surfaces is used as a partial image, there are many combinations of the two boundary surfaces (in some cases, infinite). Therefore, it is very difficult to acquire medical information in advance for all partial images included in a three-dimensional image before displaying the medical information. In contrast, in the ophthalmologic image processing device of the present disclosure, medical information is acquired by a pre-acquisition process for partial images corresponding to one or more patterns set in advance. Therefore, medical information useful to the user is presented to the user more efficiently.

[0036] Furthermore, when still images constituting a moving image are used as partial images, a pattern of a plurality of partial images may be provided in which the images are taken at regular time intervals.

[0037] The control unit may acquire, through a pre-acquisition process, medical information corresponding to a pattern selected by a user from a plurality of pre-defined patterns.

[0038] The control unit may also automatically acquire medical information corresponding to a default pattern (e.g., a two-dimensional tomographic image passing through the center of a three-dimensional tomographic image) by a pre-acquisition process. In this case, the control unit may cause the display unit to first display the medical information corresponding to the default pattern when starting to display the medical information for the acquired ophthalmic image. In this case, by setting a partial image that is likely to be selected by the user as the default pattern, user convenience is further improved. The control unit may also perform a pre-acquisition process of the medical information corresponding to the pattern before causing the display unit to display a display screen for displaying medical information for the acquired ophthalmic image. In this case, the medical information corresponding to the pattern is smoothly displayed when the display screen is displayed on the display unit.

[0039] The devices that execute the image acquisition step, medical information acquisition step, instruction receiving step, and medical information display step can be appropriately selected. For example, a control unit of a personal computer (hereinafter referred to as "PC") may execute all of the image acquisition step, medical information acquisition step, instruction receiving step, and medical information display step. That is, the control unit of the PC may acquire an ophthalmic image from an OCT device and acquire medical information based on the acquired ophthalmic image. Also, a control unit of the OCT device may execute all of the image acquisition step, medical information acquisition step, instruction receiving step, and medical information display step. Also, control units of multiple devices (e.g., OCT device and PC, etc.) may cooperate to execute the image acquisition step, medical information acquisition step, instruction receiving step, and medical information display step.

[0040] In addition, the medical information obtained from the base image by utilizing the mathematical model is not limited to a high-quality image obtained by improving the quality of the base image. For example, the mathematical model may input the base image and output an analysis result of the tissue structure of the subject's eye. In detail, the medical information may be an analysis result (segmentation result) of tissue layers and boundaries, etc. In addition, the medical information may be an analysis result regarding a disease of the subject's eye.

[0041] In addition, the ophthalmic images are not limited to those captured by an OCT device. For example, at least a part of the technology exemplified in the present disclosure can be applied to ophthalmic images captured by a scanning laser ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, a corneal endothelial cell imager (CEM), or the like.

[0042] <Embodiment> (Device configuration) Hereinafter, one of the typical embodiments of the present disclosure will be described with reference to the drawings. As shown in FIG. 1, in this embodiment, a mathematical model construction device 1, an ophthalmic image processing device 21, and ophthalmic image capturing devices 11A and 11B are used. The mathematical model construction device 1 constructs a mathematical model by training the mathematical model using a machine learning algorithm. A program for realizing the constructed mathematical model is stored in a storage device 24 of the ophthalmic image processing device 21. The ophthalmic image processing device 21 inputs an image (base image) to the mathematical model to obtain medical information. As an example, in this embodiment, a high-quality image in which the image quality of the base image is improved (for example, the effect of noise in the base image is suppressed) is obtained as medical information. The ophthalmic image capturing devices 11A and 11B capture ophthalmic images, which are images of tissues of the subject's eye.

[0043] As an example, a personal computer (hereinafter referred to as "PC") is used as the mathematical model construction device 1 of this embodiment. Although details will be described later, the mathematical model construction device 1 constructs a mathematical model by training the mathematical model using an image (hereinafter referred to as "training ophthalmic image") acquired from the ophthalmic image capturing device 11A and medical information corresponding to the training ophthalmic image. However, a device that can function as the mathematical model construction device 1 is not limited to a PC. For example, the ophthalmic image capturing device 11A may function as the mathematical model construction device 1. Furthermore, control units of a plurality of devices (for example, the CPU of the PC and the CPU 13A of the ophthalmic image capturing device 11A) may cooperate to construct a mathematical model.

[0044] In addition, a PC is used as the ophthalmic image processing device 21 of this embodiment. However, a device capable of functioning as the ophthalmic image processing device 21 is not limited to a PC. For example, the ophthalmic image capturing device 11B or a server or the like may function as the ophthalmic image processing device 21. When the ophthalmic image capturing device (OCT device in this embodiment) 11B functions as the ophthalmic image processing device 21, the ophthalmic image capturing device 11B can appropriately acquire medical information regarding a partial image included in the captured ophthalmic image while capturing an ophthalmic image. In addition, a mobile terminal such as a tablet terminal or a smartphone may function as the ophthalmic image processing device 21. Control units of a plurality of devices (for example, the CPU of the PC and the CPU 13B of the ophthalmic image capturing device 11B) may cooperate to perform various processes.

[0045] In addition, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than a CPU may be used for at least a part of the various devices. For example, a GPU may be used as a controller to speed up the processing.

[0046] The mathematical model construction device 1 will be described. The mathematical model construction device 1 is disposed, for example, at a manufacturer that provides an ophthalmic image processing device 21 or an ophthalmic image processing program to a user. The mathematical model construction device 1 includes a control unit 2 that performs various control processes, and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller that manages control, and a storage device 4 that can store programs, data, and the like. The storage device 4 stores a mathematical model construction program for executing a mathematical model construction process (see FIG. 4) described later. The communication I / F 5 also connects the mathematical model construction device 1 to other devices (for example, an ophthalmic image capturing device 11A and an ophthalmic image processing device 21, etc.).

[0047] The mathematical model construction device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the mathematical model construction device 1. For example, at least one of a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 7. Note that a microphone or the like for inputting various instructions may be used together with or instead of the operation unit 7. The display device 8 displays various images. For the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used. Note that, in this disclosure, "image" includes both still images and moving images.

[0048] The mathematical model construction device 1 can acquire data of ophthalmic images (hereinafter, may be simply referred to as "ophthalmic images") from the ophthalmic image capturing device 11A. The mathematical model construction device 1 may acquire data of ophthalmic images from the ophthalmic image capturing device 11A by at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), and the like.

[0049] The ophthalmic image processing device 21 will be described. The ophthalmic image processing device 21 is disposed in, for example, a facility (for example, a hospital or a medical examination facility) where a diagnosis or examination is performed on a subject. The ophthalmic image processing device 21 includes a control unit 22 that performs various control processes, and a communication I / F 25. The control unit 22 includes a CPU 23 that is a controller that manages control, and a storage device 24 that can store programs, data, and the like. The storage device 24 stores an ophthalmic image processing program for executing ophthalmic image processing (see FIG. 5) described later. The ophthalmic image processing program includes a program for realizing a mathematical model constructed by the mathematical model construction device 1. The communication I / F 25 connects the ophthalmic image processing device 21 to other devices (for example, the ophthalmic image capturing device 11B and the mathematical model construction device 1, etc.).

[0050] The ophthalmologic image processing device 21 is connected to an operation unit 27 and a display device 28. As with the operation unit 7 and the display device 8 described above, various devices can be used for the operation unit 27 and the display device 28.

[0051] The ophthalmological image processing device 21 can acquire ophthalmological images from the ophthalmological image capturing device 11B. The ophthalmological image processing device 21 may acquire ophthalmological images from the ophthalmological image capturing device 11B by at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), etc. The ophthalmological image processing device 21 may also acquire a program or the like that realizes the mathematical model constructed by the mathematical model construction device 1 via communication or the like.

[0052] The ophthalmic image capturing devices 11A and 11B will be described. As an example, in this embodiment, a case will be described in which an ophthalmic image capturing device 11A that provides ophthalmic images to the mathematical model construction device 1 and an ophthalmic image capturing device 11B that provides ophthalmic images to the ophthalmic image processing device 21 are used. However, the number of ophthalmic image capturing devices used is not limited to two. For example, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from a plurality of ophthalmic image capturing devices. Also, the mathematical model construction device 1 and the ophthalmic image processing device 21 may acquire ophthalmic images from one common ophthalmic image capturing device.

[0053] In the present embodiment, an OCT device is exemplified as the ophthalmic imaging device 11 (11A, 11B). However, an ophthalmic imaging device other than an OCT device (for example, a scanning laser ophthalmoscope (SLO), a fundus camera, a Scheimpflug camera, or a corneal endothelial cell imaging device (CEM)) may be used.

[0054] The ophthalmic image capturing device 11 (11A, 11B) includes a control unit 12 (12A, 12B) that performs various control processes, and an ophthalmic image capturing section 16 (16A, 16B). The control unit 12 includes a CPU 13 (13A, 13B) that is a controller that manages control, and a storage device 14 (14A, 14B) that can store programs, data, and the like. When the ophthalmic image capturing device 11 executes at least a part of the ophthalmic image processing (see FIG. 5) described below, it goes without saying that at least a part of the ophthalmic image processing program for executing the ophthalmic image processing is stored in the storage device 14.

[0055] The ophthalmologic image capturing unit 16 includes various components necessary for capturing an ophthalmologic image of the subject's eye. The ophthalmologic image capturing unit 16 of this embodiment includes an OCT light source, a branching optical element that branches the OCT light emitted from the OCT light source into measurement light and reference light, a scanning unit that scans the measurement light, an optical system that irradiates the subject's eye with the measurement light, and a light receiving element that receives a composite light of the light reflected by the tissue of the subject's eye and the reference light.

[0056] The ophthalmologic image capturing device 11 can capture two-dimensional tomographic images and three-dimensional tomographic images of the fundus of the subject's eye. In detail, the CPU 13 captures two-dimensional tomographic images (see FIG. 2, etc.) of a cross section intersecting the scan line by scanning the OCT light (measurement light) on the scan line. The CPU 13 can also capture three-dimensional tomographic images (see FIG. 3, etc.) of tissue by two-dimensionally scanning the OCT light. For example, the CPU 13 acquires multiple two-dimensional tomographic images by scanning the measurement light on each of multiple scan lines at different positions within a two-dimensional area when the tissue is viewed from the front. Next, the CPU 13 acquires a three-dimensional tomographic image by combining the captured multiple two-dimensional tomographic images.

[0057] Furthermore, the CPU 13 can also capture multiple ophthalmic images of the same part by scanning the measurement light multiple times on the same part of the tissue (on the same scan line in this embodiment). The CPU 13 can obtain an averaged image in which the influence of speckle noise is suppressed by performing an averaging process on multiple ophthalmic images of the same part. The image quality of the two-dimensional tomographic image can be improved by performing an averaging process on multiple two-dimensional tomographic images of the same part. In addition, the image quality of the three-dimensional tomographic image can be improved by making each of the multiple two-dimensional tomographic images that constitute the three-dimensional tomographic image into an averaged image. The averaging process may be performed, for example, by averaging pixel values ​​of pixels at the same position among the multiple ophthalmic images. The more images are subjected to the averaging process, the easier it is to suppress the influence of speckle noise, but the longer the imaging time. The ophthalmic image capturing device 11 performs a tracking process to make the scanning position of the OCT light follow the movement of the subject's eye while capturing multiple ophthalmic images of the same part.

[0058] (Mathematical model construction process) 2 to 4, a description will be given of a mathematical model construction process executed by the mathematical model construction device 1. The mathematical model construction process is executed by the CPU 3 in accordance with a mathematical model construction program stored in the storage device 4.

[0059] In the mathematical model construction process, a mathematical model that outputs medical information based on an ophthalmic image is constructed by training the mathematical model using a training dataset. The training dataset includes input data (input training data) and output data (output training data). The mathematical model is capable of outputting various types of medical information. The type of training dataset used to train the mathematical model is determined according to the type of medical information to be output by the mathematical model. An example of the relationship between the type of medical information to be output by the mathematical model and the training dataset used to train the mathematical model will be described below.

[0060] First, a case will be described in which a two-dimensional tomographic image is inputted as a base image into a mathematical model, and a two-dimensional tomographic image (high-quality image) in which the quality of the base image is improved is outputted to the mathematical model as medical information. In this case, in the present embodiment, a two-dimensional tomographic image of the tissue of the subject's eye is used as input training data, and a two-dimensional tomographic image of the same part, which has a higher quality than the input training data, is used as output training data to train the mathematical model. Note that the high-quality image indicates, for example, at least one of an image in which the noise of the input base image is reduced, an image in which the resolution of the original image is increased, and an image in which the visibility of the original image is improved.

[0061] FIG. 2 shows an example of input training data and output training data when a high-quality two-dimensional tomographic image is output to a mathematical model. In the example shown in FIG. 2, the CPU 3 acquires a set 40 of a plurality of two-dimensional tomographic images 400A-400X captured at the same tissue site. The CPU 3 uses a portion of the plurality of two-dimensional tomographic images 400A-400X in the set 40 (a number less than the number used for averaging the output training data described later) as input training data. The CPU 3 also acquires an average image 41 of the plurality of two-dimensional tomographic images 400A-400X in the set 40 as output training data. When a mathematical model is trained using the input training data and output training data shown in FIG. 2, a two-dimensional tomographic image is input to the trained mathematical model as a base image, and a high-quality image in which the influence of speckle noise is suppressed is output.

[0062] Next, a case will be described in which an Enface image (high-quality image) with improved image quality of the original image is output to the mathematical model by inputting the Enface image as a base image to the mathematical model. In this case, in the present embodiment, the Enface image of the tissue of the subject eye is used as input training data, and the Enface image of the same part, which has higher image quality than the input training data, is used as output training data to train the mathematical model.

[0063] An example of the enface image 55 will be described with reference to FIG. 3. The enface image 55 is a two-dimensional image of a part of a three-dimensional tomographic image (partial image) included in the entire image area of ​​the three-dimensional tomographic image 50, viewed from a direction (front direction) along the optical axis of the measurement light of the ophthalmic imaging device (OCT device) 11. The enface image 55 may be, for example, an integrated image in which pixel values ​​of the partial image are integrated in the depth direction (Z direction) at each position in the XY direction intersecting the optical axis. The enface image 55 may also be generated by an integrated value of the spectral data of the partial image at each position in the XY direction. In this embodiment, the three-dimensional tomographic image sandwiched between two boundary surfaces (slabs) 52A and 52B in the image area of ​​the three-dimensional tomographic image 50 is treated as a partial image, and the enface image 55 of the partial image is generated. 3, the positions of two boundary surfaces 52A and 52B are illustrated on one two-dimensional tomographic image 51 (for example, a two-dimensional tomographic image passing through the center of the three-dimensional tomographic image 50) constituting the three-dimensional tomographic image 50. When the positions of the boundary surfaces 52A and 52B are changed, the partial image and the Enface image 55 generated from the partial image are also changed.

[0064] An example of input training data and output training data when a high-quality Enface image 55 is output to a mathematical model will be described. The CPU 3 acquires a plurality of three-dimensional tomographic images captured by the fundus image capturing device 11A of the same tissue (the fundus of the same subject eye in this embodiment). The plurality of three-dimensional tomographic images include a low-quality three-dimensional tomographic image and a high-quality three-dimensional tomographic image. As described above, the high-quality three-dimensional tomographic image may be acquired by averaging each of a plurality of two-dimensional tomographic images that constitute the image. Next, the CPU 3 extracts a portion of the three-dimensional tomographic image as a partial image from the low-quality three-dimensional tomographic image, and sets the Enface image generated from the extracted partial image as the input training data. The CPU 3 extracts a three-dimensional tomographic image of the same range as the input training data as a partial image from the high-quality three-dimensional tomographic image, and sets the Enface image generated from the extracted partial image as the output training data. When a mathematical model is trained using input training data and output training data, an Enface image is input as a base image to the trained mathematical model, and a high-quality image with improved image quality of the base image is output.

[0065] It is also possible to change the method of generating high-quality two-dimensional or three-dimensional tomographic images. For example, the image quality may be improved by a process other than the averaging process. Furthermore, the input training data is not limited to tomographic images of tissues.

[0066] The mathematical model may also be trained to output medical information other than the high-quality image. For example, by inputting an ophthalmic image to the mathematical model, it is also possible to cause the mathematical model to output an analysis result of the structure of the tissue of the subject's eye as medical information. In this case, the mathematical model is trained using an ophthalmic image of the tissue of the subject's eye as input training data and medical data indicating the structure of the tissue reflected in the input training data (ophthalmic image) as output training data. As an example, the mathematical model may be trained using a two-dimensional tomographic image of the fundus tissue as input training data and data indicating the position of the structure of the tissue reflected in the input training data (e.g., at least one of the positions of the layer and the boundary of the tissue) as output training data. In this case, when the two-dimensional tomographic image is input to the trained mathematical model, the analysis result of the structure of the tissue (e.g., at least one of the positions of the layer and the boundary) is output. It is also possible to cause the mathematical model to output at least one of the analysis results of the fundus blood vessels, the fovea, and the optic disc of the subject's eye. The input training data does not need to be a two-dimensional tomographic image, and may be a two-dimensional front image or the like.

[0067] In addition, by inputting an ophthalmic image into the mathematical model, it is possible to output an automatic analysis result indicating whether or not the subject's eye has any disease to the mathematical model as medical information. In this case, the mathematical model may be trained using an ophthalmic image of the tissue of the subject's eye (for example, at least one of a two-dimensional tomographic image, a three-dimensional tomographic image, an Enface image, etc.) as input training data, and data indicating at least one of the presence or absence of a disease and the type of the disease in the subject captured in the input training data as output training data.

[0068] The mathematical model construction process will be described with reference to Fig. 4. The CPU 3 acquires at least a part of an ophthalmic image captured by the ophthalmic image capturing device 11A as input training data (S1). In this embodiment, data of the ophthalmic image is generated by the ophthalmic image capturing device 11A and then acquired by the mathematical model construction device 1. However, the CPU 3 may acquire data of the ophthalmic image by acquiring a signal (e.g., an OCT signal) that is the basis for generating the ophthalmic image from the ophthalmic image capturing device 11A and generating the ophthalmic image based on the acquired signal.

[0069] Next, the CPU 3 acquires output training data corresponding to the input training data acquired in S1 (S3). An example of the correspondence between the input training data and the output training data is as described above.

[0070] Next, the CPU 3 executes training of the mathematical model using the training data set by a machine learning algorithm (S3). Commonly known machine learning algorithms include, for example, neural networks, random forests, boosting, and support vector machines (SVMs).

[0071] Neural networks are a method to mimic the behavior of biological neural networks. Examples of neural networks include feedforward neural networks, radial basis function (RBF) networks, spiking neural networks, convolutional neural networks, recurrent neural networks (recurrent neural networks, feedback neural networks, etc.), and probabilistic neural networks (Boltzmann machines, Bayesian networks, etc.).

[0072] Random forest is a method to generate a large number of decision trees by learning based on randomly sampled training data. When using random forest, the branches of multiple decision trees that have been trained in advance as classifiers are traced, and the average (or majority vote) of the results obtained from each decision tree is taken.

[0073] Boosting is a method to generate a strong classifier by combining multiple weak classifiers. A strong classifier is constructed by sequentially training simple weak classifiers.

[0074] SVM is a method for constructing a two-class pattern classifier using linear input elements. For example, SVM learns the parameters of the linear input elements based on the criterion of finding the margin-maximizing hyperplane that maximizes the distance from each data point from the training data (hyperplane separation theorem).

[0075] A mathematical model refers to, for example, a data structure for predicting the relationship between input data and output data. A mathematical model is constructed by being trained using a training data set. As described above, a training data set is a set of input training data and output training data. For example, the correlation data (e.g., weights) between each input and output is updated by training.

[0076] In this embodiment, a multi-layered neural network is used as the machine learning algorithm. The neural network includes an input layer for inputting data, an output layer for generating data to be predicted, and one or more hidden layers between the input layer and the output layer. A plurality of nodes (also called units) are arranged in each layer. In detail, in this embodiment, a convolutional neural network (CNN), which is a type of multi-layered neural network, is used. However, other machine learning algorithms may be used. For example, generative adversarial networks (GAN), which utilize two competing neural networks, may be adopted as the machine learning algorithm.

[0077] The processes of S1 to S3 are repeated until the construction of the mathematical model is completed (S5: NO). When the construction of the mathematical model is completed (S5: YES), the mathematical model construction process ends. The program and data for realizing the constructed mathematical model are incorporated into the ophthalmologic image-processing device 21.

[0078] (Ophthalmology Image Processing) An example of ophthalmological image processing executed by the ophthalmological image processing device 21 will be described with reference to Fig. 5 to Fig. 11. Fig. 5 to Fig. 11 illustrate a case where a plurality of two-dimensional tomographic images constituting a three-dimensional tomographic image of the fundus are treated as partial images, and a high-quality image in which the quality of each partial image is improved is acquired and displayed. The ophthalmological image processing illustrated in Fig. 5 is executed by the CPU 23 according to an ophthalmological image processing program stored in the storage device 24.

[0079] 5, the CPU 23 acquires an ophthalmic image of the tissue of the subject's eye captured by an ophthalmic image capturing device (an OCT device in this embodiment) 11B (S11). In this embodiment, in S11, a three-dimensional tomographic image of the fundus tissue of the subject's eye (for example, see the three-dimensional tomographic image 50 shown in FIG. 3) is acquired. As described above, the three-dimensional tomographic image is generated by combining (arranging) a plurality of two-dimensional tomographic images.

[0080] Next, the CPU 23 causes the display device 28 to display a partial image selection image 61 (S12). Fig. 6 shows an example of a display screen 60 displayed on the display device 28 during execution of ophthalmologic image processing. The display screen 60 displays the partial image selection image 61 and a selected image display field 65.

[0081] The partial image selection image 61 is displayed so that the user can select at least one of a plurality of partial images 66 (two-dimensional tomographic images in this embodiment) included in the three-dimensional tomographic image. In the example shown in FIG. 6, the partial image selection image 61 includes a two-dimensional front image when the three-dimensional tomographic image is viewed from the optical axis direction of the OCT measurement light. A selection range 62 in which the partial image 66 can be selected is displayed in the image area of ​​the two-dimensional front image. A selection line 63 through which the selected partial image 66 passes is displayed in the selection range 62. The user can input an instruction to select the partial image 66 to the ophthalmologic image processing device 21 by operating the operation unit 27 (e.g., a mouse, etc.) and setting (e.g., moving) the position of the selection line 63 to a desired position. The CPU 23 determines the partial image 66 at the position through which the selection line 63 passes as the partial image 66 selected by the user.

[0082] In the selected image display field 65, at least one of the partial image 66 selected by the user and a high-quality image 67 (see FIG. 7) related to the selected partial image 66 is displayed. In the example shown in FIG. 6, since an inference process for acquiring a high-quality image 67 related to the partial image 66 is in progress, the partial image 66 itself is displayed. When acquisition of the high-quality image 67 is completed, the high-quality image 67 is displayed instead of the partial image 66. In addition, if the high-quality image 67 related to the selected partial image 66 has already been acquired in a pre-acquisition process (see FIG. 11) described later, the high-quality image 67 related to the selected partial image 66 is displayed in the selected image display field 65 at the time the partial image 66 is selected. Thus, the user's operation time is appropriately shortened.

[0083] Moreover, the display screen 60 (in the present embodiment, the selected image display field 65 in the display screen 60) includes partial image switching buttons 68F and 68B. The partial image switching button 68F is operated to change (switch) the partial image 66 to be selected from the partial image 66 selected at that time to the partial image 66 adjacent in the forward direction (for example, in the example shown in FIG. 6, the downward direction of the screen in the selection range 62). The partial image switching button 68B is operated to change the partial image 66 to be selected from the partial image 66 selected at that time to the partial image 66 adjacent in the reverse direction (for example, in the example shown in FIG. 6, the upward direction of the screen in the selection range 62). The user can switch the partial images 66 in a predetermined direction by successively operating the partial image switching button 68F or the partial image switching button 68B.

[0084] Furthermore, an image type display section 69 is displayed on the display screen 60. The image type display section 69 displays whether the image currently displayed in the selected image display section 65 is the partial image 66 selected by the user or the high-quality image 67 related to the partial image 66. Therefore, the user can easily know whether the image displayed in the selected image display section 65 is the high-quality image 67 or not.

[0085] In this embodiment, a thickness map showing a two-dimensional distribution of the thickness of a specific layer in the fundus tissue (a layer between the ILM and the RPE / BM in the example shown in FIG. 6) is superimposed on the selection range 62 in the partial image selection image 61. Therefore, the user can select a partial image 66 after considering the distribution of the thickness of a specific layer in the fundus.

[0086] Furthermore, when the CPU 23 starts displaying the display screen 60 in S12, it automatically displays a high-quality image 67 relating to a partial image 66 of a default pattern (for example, a partial image 66 passing through the center of an image area of ​​a two-dimensional front image) in the selected image display field 65 first. Here, the CPU 23 inputs the partial image 66 of the default pattern into a mathematical model before displaying the display screen 60, thereby acquiring in advance the high-quality image 67 relating to the partial image 66 of the default pattern. The process of acquiring in advance the high-quality image 67 corresponding to the default pattern is also an example of a pre-acquisition process. When the CPU 23 displays the display screen 60, it displays the high-quality image 67 corresponding to the default pattern in the selected image display field 65 from the beginning. Therefore, the high-quality image 67 corresponding to the default pattern is displayed smoothly.

[0087] Returning to the explanation of Fig. 5, the CPU 23 judges whether or not a pre-acquisition pattern has been set by the user (S13). Fig. 8 shows an example of a setting screen displayed on the display device 28 when the user sets a pre-acquisition pattern. The pre-acquisition pattern is a pattern of partial images 66 from which medical information (high-quality images 67 in this embodiment) is acquired in advance, among a plurality of partial images 66 in an ophthalmologic image (three-dimensional tomographic image in this embodiment).

[0088] 8, when an instruction to set a pre-acquired pattern is input by the user, the CPU 23 causes the display device 28 to display a pattern selection field 70, a position designation field 71, an angle designation field 72, and a size designation field 73. In the pattern selection field 70, a plurality of pre-acquired patterns that differ from each other in at least one of the arrangement, number, and shape of the lines through which the partial image 66 passes are displayed. The user can arbitrarily select one or more of the plurality of pre-acquired patterns.

[0089] In the position designation field 71, a position on the partial image selection image 61 (see FIG. 6) where the selected pre-acquired pattern is to be set is designated. For example, when "macular center" is designated, the position of the pre-acquired pattern is set so that the center of the selected pre-acquired pattern coincides with the macula (for example, the fovea, which is the center of the macula). The positions of the macula, optic disc, abnormal area, and the like in the ophthalmic image may be detected by performing image processing on the ophthalmic image. In addition, the positions of the macula, and the like may be acquired by utilizing a mathematical model trained by a machine learning algorithm. When "manual setting" is designated, the selected pre-acquired pattern is set at a position designated by an instruction from the user. In the angle designation field 72, the angle of the selected pre-acquired pattern on the ophthalmic image is designated. In the size designation field 90, the size of the selected pre-acquired pattern on the ophthalmic image is set.

[0090] Returning to the explanation of FIG. 5, when a pre-acquired pattern is set (S13: YES), the CPU 23 inputs a partial image 66 corresponding to the set pre-acquired pattern into a mathematical model, thereby acquiring a high-quality image 67 in which the quality of the input partial image 66 is improved, as medical information. The process of acquiring the high-quality image 67 corresponding to the pre-acquired pattern is an example of a pre-acquiring process. The CPU 23 stores the acquired high-quality image 67 in the storage device 24 and displays it on the display device 28 (S14). Thus, the user can properly check the high-quality image 67 corresponding to the desired pattern.

[0091] Note that the high-quality image 67 corresponding to the pre-acquired pattern may be executed in advance before the process (S12) of starting the display on the display screen 60. In this case, in S12, the high-quality image 67 corresponding to the pre-acquired pattern may be displayed in the selected image display field 65 from the beginning. As a result, the high-quality image 67 corresponding to the pre-acquired pattern is displayed on the display device 28 more smoothly.

[0092] Next, the CPU 23 judges whether or not follow-up position information is stored in the storage device 24 (S16). The follow-up position information is information indicating the position of a partial image 66 selected by the user when ophthalmic image processing was previously performed on an ophthalmic image of the same photographing subject (i.e., the position of a reference image, which will be described later). If the follow-up position information is stored (S16: YES), the CPU 23 sets the partial image 66 at the follow-up position as the reference image. The reference image will be described later. If a plurality of follow-up positions are stored, any of the follow-up positions (for example, the position of the partial image 66 first selected by the user in the past processing, etc.) may be adopted. Furthermore, if the follow-up position information is stored, the partial image 66 at the follow-up position may be set as a default partial image 66 that initially displays a high-quality image 67 on the display screen 60.

[0093] If the follow-up position information is not stored (S16: NO), the CPU 23 acquires a specific position of the tissue shown in the ophthalmologic image (S18). The specific position may be, for example, at least one of the macula, the optic disc, the lesion, and the fundus blood vessels in the fundus. The CPU 23 may detect the specific position by, for example, performing image processing on the ophthalmologic image. The CPU 23 may also acquire the specific position by using a mathematical model trained by a machine learning algorithm. The CPU 23 sets the partial image 66 at the specific position to a reference image (S19) to be described later. However, the specific position may be determined in advance. For example, the position of the partial image 66 of the above-mentioned default pattern may be set as the specific position.

[0094] Next, the CPU 23 sets a priority order for performing a pre-acquisition process (see FIG. 9), which will be described later, for each of the multiple partial images 66 included in the ophthalmologic image, based on the reference image (S21). The priority order indicates the order in which the pre-acquisition process is performed for each of the multiple partial images 66. In other words, the process of acquiring medical information (high-quality image 66 in this embodiment) for each partial image 66 is executed in order according to the priority order. The reference image is a partial image 66 that serves as a standard for setting the priority order for the multiple partial images 66.

[0095] In S21 of this embodiment, the CPU 23 sets priorities of the multiple partial images 66 such that the closer the partial images are to the reference image, the higher the priority. In detail, in S21 of this embodiment, the CPU 23 sets priorities of the multiple partial images 66 such that the closer the partial images are to the reference image, the higher the priority.

[0096] FIG. 9 is an explanatory diagram for explaining an example of a method for setting priorities for a plurality of partial images 66 with reference to a reference image BL. FIG. 9 shows a state in which a line through which each of a plurality of partial images 66 passes is viewed from a direction along the optical axis of the OCT measurement light. As shown in FIG. 9, in S21 of this embodiment, the CPU 23 sets the priority of two partial images 66 adjacent to each other in the forward direction (downward in the figure) and the reverse direction (upward in the figure) of the reference image BL to the highest priority P1. In addition, the CPU 23 sets the priority such that the priority decreases in the order of P2, P3, in both the forward direction and the reverse direction as the distance from the reference image BL increases. As a result, medical information related to a partial image 66 closer to the reference image BL is preferentially acquired.

[0097] Returning to the description of Fig. 5, the CPU 23 starts the advance acquisition process (S22). In the advance acquisition process, medical information related to the partial image 66 that has not yet been selected by the user is acquired in advance and stored. Details of the advance acquisition process will be described later with reference to Fig. 11.

[0098] Next, the CPU 23 judges whether or not an instruction to select a partial image 66 has been input by the user (S24). If no instruction has been input (S24: NO), the process proceeds directly to S29. If the CPU 23 accepts an input of an instruction to select a partial image 66 (S24: YES), the CPU 23 sets the selected partial image 66 as a new reference image (S25). The CPU 23 stores information on the position of the newly set reference image in the storage device 24 as follow-up position information to be used when performing ophthalmic image processing on an ophthalmic image of the same examinee's eye in the future (S26). The CPU 23 resets the priority order for the multiple partial images 66 in order of proximity to the newly set reference image (S27).

[0099] In S27, when the partial image 66 selected by the user is changed, the CPU 23 can set the priority order for the multiple partial images 66 in order along the direction of the changed partial image 66 relative to the partial image 66 before the change. For example, when the partial image 66 to be selected is switched by operating the partial image switching button 66F or the partial image switching button 66B (see FIG. 6), the user often switches the selected partial image 66 continuously along the same direction (forward or backward). Therefore, when the partial image 66 is switched by the partial image switching button 66F or the partial image switching button 66B in this embodiment, the CPU 23 resets the priority order along the direction in which the partial image 66 is switched.

[0100] Fig. 10 is an explanatory diagram for explaining an example of a method for setting priorities for a plurality of partial images 66 along the direction in which the partial image 66 selected by the user is switched. In the example shown in Fig. 10, the partial image 66 selected by the user is switched from BL1 to BL2. In this case, the CPU 23 changes the reference image from BL1 to BL2 (S25), and resets the priorities along the direction of the partial image 66 (BL2) after the change relative to the partial image 66 (BL1) before the change (the downward direction in the drawing, which is the forward direction, in the example shown in Fig. 10). That is, the priorities are set so that the priorities decrease in the order of P1, P2, and P3 as the distance from the reference image BL2 increases in the changed direction.

[0101] Next, the CPU 23 displays medical information (high-quality image 67 in this embodiment) related to the partial image 66 selected in S24 on the display device 28 (selected image display field 65 shown in FIG. 6 in this embodiment) (S28). In detail, when medical information related to the selected partial image 66 has not yet been acquired, the CPU 23 inputs the selected partial image 66 into a mathematical model, thereby acquiring and displaying medical information related to the input partial image 66. As described above, while the process of acquiring medical information is being executed, the selected partial image 66 is displayed in the selected image display field 65. When medical information related to the selected partial image 66 has already been acquired and stored in the storage device 24, the CPU 23 displays the stored medical information on the display device 24.

[0102] The processes in S24 to S29 are repeated until an instruction to end the process is input (S29: NO), When an instruction to end the process is input (S29: YES), the ophthalmologic image process is ended.

[0103] An example of the pre-acquisition process executed by the ophthalmological image processing device 21 will be described with reference to Fig. 11. As described above, in the pre-acquisition process, medical information on partial images 66 that have not yet been selected by the user is acquired and stored in advance. The acquisition process of medical information on each partial image 66 is executed in order according to a set priority order. The pre-acquisition process illustrated in Fig. 11 is executed by the CPU 23 according to an ophthalmological image processing program stored in the storage device 24. The pre-acquisition process is executed in the background in parallel with the ophthalmological image processing illustrated in Fig. 5.

[0104] First, the CPU 23 sets the value of a counter N for counting the priority to "1" (S31). The CPU 23 determines whether or not medical information on the partial image 66 with the Nth priority has already been acquired (S32). If the medical information on the Nth partial image 66 has already been stored in the storage device 24 (S32: YES), in order to avoid duplication of processing and shorten the processing time, the CPU 23 adds "1" to the value of the counter N (S33). The process returns to S32. In other words, if the medical information on the Nth partial image 66 has already been acquired, the process of acquiring the medical information is omitted.

[0105] If medical information on the Nth partial image 66 has not yet been acquired (S32: NO), the CPU 23 sets the partial image 66 with the Nth priority as a base image and inputs it as the base image to the mathematical model trained by the machine learning algorithm. As a result, inference processing using the mathematical model is started to output a high-quality image 67 (see FIG. 7) in which the quality of the base image is improved as medical information (S34).

[0106] The CPU 23 judges whether or not a new reference image has been set in the ophthalmologic image processing (see FIG. 5) (S35). If a new reference image has not been set (S35: NO), it is judged whether or not the medical information acquisition process for the Nth partial image 66 (i.e., the inference process started in S34) has been completed (S36). If the medical information acquisition process has not been completed (S36: NO), the process returns to S35, and the processes of S35 and S36 are repeated.

[0107] When the medical information acquisition process for the Nth partial image 66 is completed (S36: YES), the CPU 23 stores the acquired medical information in the storage device 24 (S38). The CPU 23 adds "1" to the value of the counter N for counting the priority order (S39). The CPU 24 determines whether or not a series of medical information acquisition processes for the multiple partial images 66 for which priority order has been set in the ophthalmologic image processing (see FIG. 5) (i.e., the medical information acquisition process for the group of partial images 66 that has been executed using the current counter N) has been completed (S40). If not completed (S40: NO), the process returns to S32, and the process for the partial image 40 for which the next priority order has been set is performed (S32 to S38).

[0108] When a new reference image is set in the ophthalmologic image processing (see FIG. 5) (S35: YES) before the process of acquiring medical information for the Nth partial image 66 is completed (S36: NO), it is desirable to change the priority order for acquiring medical information to the priority order based on the newly set reference image. Therefore, the CPU 23 interrupts the pre-acquisition process (inference process) being executed (S46). The priority order reset in S27 is set again based on the newly set reference image (S47). The process returns to S31, and the pre-acquisition process (S31 to S40) is executed according to the newly set priority order. That is, the CPU 23 interrupts the process being executed for the group of partial images 66, and resumes the process for the new group of partial images 66.

[0109] When the process of acquiring medical information for the group of partial images 66 that was being executed using the current counter N is completed (S40: YES), the CPU 23 determines whether or not there is any process for another group of partial images 66 that was suspended in S46 (S41). If there is any suspended process (S41: YES), the CPU 23 resets the priority of the suspended process and the counter N (S42), and the process returns to S32. As a result, the suspended process is appropriately resumed.

[0110] If there is no interrupted process (S41: NO), it is determined whether the process of acquiring medical information for all partial images 66 included in the ophthalmologic image is completed (S44). If not completed (S44: NO), a new priority is set (S49) to execute the process for the partial images 66 for which medical information has not yet been acquired, and the process returns to S31. When medical information for all partial images 66 has been acquired (S44: YES), the pre-acquisition process ends.

[0111] (First transformation example) A first modification of the above embodiment will be described with reference to FIG. 12. FIG. 12 is an example of a display screen 80 displayed on the display device 28 during the execution of ophthalmic image processing in the first modification. The ophthalmic image processing device 1 according to the first modification takes a part of a three-dimensional tomographic image included in the entire image area of ​​the three-dimensional tomographic image as a partial image, and generates an Enface image 55 from the partial image. The ophthalmic image processing device 1 acquires a high-quality image 85 (85A, 85B, 85C) in which the quality of the generated Enface image 55 is improved, as medical information. At least a part of the ophthalmic image processing (see FIG. 5) and the advance acquisition processing (see FIG. 11) exemplified in the above embodiment can be similarly adopted in the following first modification. Therefore, the description of the processing in which the same processing as the above embodiment can be adopted will be omitted or simplified.

[0112] In the ophthalmic image processing according to the first modification (see FIG. 5), first, a three-dimensional tomographic image of the fundus tissue of the subject eye (for example, see the three-dimensional tomographic image 50 shown in FIG. 3) is acquired (S11). Next, the CPU 23 causes the display device 28 to display a partial image selection image 81 (see FIG. 12) (S12). As shown in FIG. 12, during the execution of the ophthalmic image processing, the partial image selection image 81 and high-quality images 85A, 85B, and 85C are displayed on the display screen 80. For the partial image selection image 81 of the first modification, one two-dimensional tomographic image constituting the three-dimensional tomographic image acquired in S11 is used. For the partial image selection image 81, the positions of two boundary surfaces (slabs) 82A and 82B for defining the range of the partial image are displayed. The three-dimensional tomographic image in the area sandwiched between the two boundary surfaces 82A and 82B becomes the partial image in the first modification.

[0113] Internally, for each of the multiple two-dimensional tomographic images constituting the three-dimensional tomographic image acquired in S11, the positions of at least one of the tissue layers and layer boundaries (hereinafter simply referred to as "layer / boundary") appearing in the image are acquired. The positions of the layer / boundary may be acquired by image processing of the two-dimensional tomographic image, or may be acquired by using a machine learning algorithm. Alternatively, the positions of the layer / boundary may be acquired by the user specifying the positions of the layer / boundary.

[0114] The user operates the operation unit 27 to set (for example, move) at least one of the two displayed boundary surfaces 82A, 82B to a desired position. As a result, the positions of the boundary surfaces 82A, 82B that define the range of the partial image are set. That is, the user can select a partial image in the ophthalmologic image acquired in S11 by setting the positions of the boundary surfaces 82A, 82B. As an example, the CPU 23 sets two boundary surfaces for each of the multiple two-dimensional tomographic images that constitute the three-dimensional tomographic image based on the distance and direction of the boundary surfaces 82A, 82B set by the user relative to the layer-boundary positions of the already acquired tissues. The CPU 23 sets the three-dimensional tomographic image of the area sandwiched between the two set boundary surfaces as the partial image.

[0115] Returning to the description of FIG. 5, the CPU 23 judges whether or not the pre-acquired pattern is set by the user (S13). The pre-acquired pattern in the first modified example is provided with a pattern in which a specific layer that is likely to attract the user's attention is a partial image. In the example shown in FIG. 12, a pattern in which the vitreous body interface is a partial image and a pattern in which the inner retina layer is a partial image are set. The CPU 23 generates the Enface image 55 (see FIG. 3) based on the partial image corresponding to the set pre-acquired pattern. The CPU 23 inputs the generated Enface image 55 as a base image into a mathematical model to acquire high-quality images 85A and 85B. The process of acquiring the high-quality images 85A and 85B corresponding to the pre-acquired pattern is an example of a pre-acquired process. The CPU 23 stores the acquired high-quality images 85A and 85B in the storage device 24 and displays them on the display device 28 (S14).

[0116] As described above, the high-quality images 85A, 85B corresponding to the pre-acquired pattern may be executed in advance before the process (S12) of starting the display on the display screen 80. In this case, in S12, the high-quality images 85A, 85B corresponding to the pre-acquired pattern may be displayed on the display screen 80 from the beginning. As a result, the high-quality images 85A, 85B corresponding to the pre-acquired pattern are displayed on the display device 28 more smoothly.

[0117] Next, the CPU 23 determines whether or not follow-up position information is stored in the storage device 24 (S16). The follow-up position may be set based on the positions of two boundary surfaces 82A, 82B selected by the user in a previous process. In addition, the specific position used in the processes of S18 and S19 may be the position of a specific layer / boundary.

[0118] In S21 in the first modified example, the priority order of partial images is set based on at least one of the closeness of the distance, the closeness of the range, the closeness of the overlap rate, etc., relative to the three-dimensional tomographic image set as the reference image. For example, the CPU 23 may set the priority order of the partial images sandwiched between the boundary surfaces 82A, 82B in the order of proximity to the reference image when the positions of the two boundary surfaces 82A, 82B are repeatedly moved by unit distances while keeping the distance between the two boundary surfaces 82A, 82B constant. The CPU 23 may also set the priority order of the partial images sandwiched between the boundary surfaces 82A, 82B in the order of proximity to the reference image when the positions of one of the two boundary surfaces 81A, 82B are repeatedly moved by unit distances.

[0119] In addition, in S27 in the first modified example, the priority order may be set according to the direction in which the partial image was changed (i.e., the direction in which the boundary surfaces 82A, 82B were changed). In addition, in S28 in the first modified example, as shown in FIG. 12, a high-quality image 85C of the Enface image 55 based on the selected partial image is displayed on the display screen 28.

[0120] In addition, in S34 (see FIG. 11) in the first modified example, the CPU 23 generates an Enface image 55 based on the partial image with the Nth priority order. The CPU 23 inputs the generated Enface image 55 into a mathematical model to obtain a high-quality image 85C.

[0121] (Second transformation example) A second modified example of the above embodiment will be described. The ophthalmic image processing device 1 of the above embodiment and the first modified example acquires a high-quality image obtained by improving the quality of a base image as medical information. However, the medical information acquired by the ophthalmic image processing device 1 is not limited to a high-quality image. The ophthalmic image processing device 1 of the second modified example acquires an analysis result of the structure of the tissue of the subject eye as medical information by inputting a partial image or an image generated based on the partial image into a mathematical model. In detail, the medical information may be an analysis result of the layer / boundary of the tissue (segmentation result), etc. Also, the medical information may be an analysis result of a lesion site of the tissue, or a fundus blood vessel, etc. In this case, the above-mentioned training method, etc. can be adopted as a training method for the mathematical model. Also, the medical information may be an analysis result regarding a disease of the subject eye.

[0122] The techniques disclosed in the above embodiment are merely examples. Therefore, the techniques exemplified in the above embodiment can be modified. First, it is possible to execute only a part of the techniques exemplified in the above embodiment. For example, it is possible to omit at least one of the processes of S13 to S14, S16 to S17, and S18 to S19 in the ophthalmic image processing (see FIG. 5). Moreover, instead of executing the processes of S18 and S19, the CPU 23 may set a predetermined partial image (for example, a partial image including the center of the image area of ​​the ophthalmic image acquired in S11) as the reference image. Moreover, the process (S22) of starting the advance acquisition process may be executed after an instruction to select a partial image is accepted.

[0123] The process of acquiring an ophthalmologic image in S11 of Fig. 5 is an example of an "image acquiring step." The processes of acquiring medical information in S14 and S28 of Fig. 5 and S34 to S38 of Fig. 11 are an example of a "medical information acquiring step." The process of accepting an instruction to select a partial image in S24 of Fig. 5 is an example of an "instruction accepting step." The processes of displaying medical information on display device 28 in S14 and S28 of Fig. 5 is an example of a "medical information display step." [Explanation of symbols]

[0124] 11A, 11B Ophthalmic imaging device 21 Ophthalmological image processing device 23 CPU 24 Storage device 27 Control section 28 Display device 50 3D tomographic images 51 2D Tomographic Images 55 Enface Images 66 Partial Image 67 High Quality Images 82A,82B Boundary surface 85A, 85B, 85C High quality images

Claims

1. An ophthalmic image processing program executed by an ophthalmic image processing device for processing an ophthalmic image which is an image of tissue of a subject's eye, The ophthalmologic image processing program is executed by a control unit of the ophthalmologic image processing device, An image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT device; a medical information acquisition step of generating an enface image when the partial image is viewed from a direction along the optical axis of the measurement light of the OCT device based on a partial image, which is a three-dimensional tomographic image within a partial region sandwiched between two boundary surfaces, of the three-dimensional tomographic image acquired in the image acquisition step, and inputting the enface image as a base image into a mathematical model trained by a machine learning algorithm to acquire a high-quality image with improved image quality of the base image, and storing the high-quality image in a storage device; an instruction receiving step of receiving an input of an instruction to select a partial image, which is a three-dimensional tomographic image within a partial region between two boundary surfaces in the three-dimensional tomographic image acquired in the image acquiring step; a medical information display step of displaying, when an instruction to select a partial image is received, a high-quality image acquired for the partial image selected by the instruction on a display unit; is executed by the ophthalmic image processing device, In the medical information acquisition step, a pre-acquisition process is executed to acquire and store high-quality images of partial images that have not yet been selected in response to an instruction to select a partial image from among the three-dimensional tomographic images, before an input of an instruction to select the partial image is actually accepted in the instruction acceptance step; An ophthalmologic image processing program characterized in that the pre-acquisition processing is executed in parallel with the instruction receiving step of receiving input of an instruction to select a partial image, and the medical information display step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction to select the partial image.

2. 2. An ophthalmologic image processing program according to claim 1, An ophthalmologic image processing program characterized in that, in the instruction receiving step, the control unit inputs an instruction to set the position of at least one of the two boundary surfaces that define the range of the partial image as an instruction to select the partial image.

3. 3. An ophthalmologic image processing program according to claim 2, An ophthalmologic image processing program characterized in that, in the instruction receiving step, the control unit inputs an instruction to move the position of at least one of the two displayed boundary surfaces that define the range of a partial image while displaying the positions of the two boundary surfaces that define the range of a partial image on a two-dimensional tomographic image that constitutes the three-dimensional tomographic image acquired in the image acquisition step, as an instruction to select a partial image.

4. 4. An ophthalmologic image processing program according to claim 2, An ophthalmologic image processing program characterized in that, in the pre-acquisition processing, the control unit sequentially executes high-quality image acquisition processing for each partial image in order of priority set for multiple partial images of the three-dimensional tomographic image acquired in the image acquisition step, in which the positions of at least one of the two boundary surfaces differ from each other.

5. 5. An ophthalmologic image processing program according to claim 4, An ophthalmologic image processing program characterized in that, in the pre-acquisition processing, when the partial image selected by the instruction is changed, the control unit sets the priority order for the multiple partial images in order along the direction of the boundary surface of the partial image after the change relative to the boundary surface of the partial image before the change.

6. 6. An ophthalmologic image processing program according to claim 1, An ophthalmologic image processing device, wherein the Enface image is a motion contrast image obtained by processing at least two OCT signals obtained at different times for the same position.

7. An ophthalmic image processing device that processes an ophthalmic image which is an image of tissue of a subject's eye, The control unit of the ophthalmologic image processing device An image acquisition step of acquiring a three-dimensional tomographic image of the tissue of the subject's eye captured by an OCT device; a medical information acquisition step of generating an enface image when the partial image is viewed from a direction along the optical axis of the measurement light of the OCT device based on a partial image, which is a three-dimensional tomographic image within a partial region sandwiched between two boundary surfaces, of the three-dimensional tomographic image acquired in the image acquisition step, and inputting the enface image as a base image into a mathematical model trained by a machine learning algorithm to acquire a high-quality image with improved image quality of the base image, and storing the high-quality image in a storage device; an instruction receiving step of receiving an input of an instruction to select a partial image, which is a three-dimensional tomographic image within a partial region between two boundary surfaces in the acquired three-dimensional tomographic image; a medical information display step of displaying, when an instruction to select a partial image is received, a high-quality image acquired for the partial image selected by the instruction on a display unit; Run In the medical information acquisition step, the control unit executes a pre-acquisition process for acquiring and storing high-quality images of partial images that have not yet been selected by an instruction to select a partial image from among the three-dimensional tomographic images, before an input of an instruction to select the partial image is actually accepted in the instruction acceptance step; An ophthalmologic image processing device characterized in that the pre-acquisition processing is executed in parallel with the instruction receiving step of receiving input of an instruction to select a partial image, and the medical information display step of displaying on a display unit a high-quality image acquired for the partial image selected by the instruction to select the partial image.

Images