Ophthalmic image processing device and ophthalmic image processing program

Abstract

To enable a user to properly determine the position of structure information in a subject eye.SOLUTION: One of an ophthalmologic image processing device disclosed herein is the ophthalmologic image processing device for processing an ophthalmologic image being the image of a tissue of a subject eye. A control unit of the ophthalmologic image processing device acquires an ophthalmologic image captured by an ophthalmologic image capturing device, acquires the probability distribution for identifying the tissue in the ophthalmologic image by inputting the ophthalmologic image to a mathematical model trained with a machine learning algorithm, acquires the alienation of the acquired probability distribution to the probability distribution in a case where the tissue is correctly identified as structure information indicating an abnormality degree of the structure of the tissue, causes display means to display the image of the subject eye, and displays the structure information indicating the abnormality degree of the structure of the tissue so as to be overlapped on the image of the subject eye.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosed technology relates to an ophthalmological image processing device and an ophthalmological image processing program. [Background technology]

[0002] Patent Document 1 discloses a technology for acquiring a probability distribution for identifying tissues in an ophthalmic image by inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm. Patent Document 1 also discloses a technology for allowing a user to appropriately determine abnormalities in the structure of tissues shown in an ophthalmic image by acquiring structural information indicating the degree of abnormality in the structure of the tissue using the acquired probability distribution. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-18794 Summary of the Invention [Problem to be solved by the invention]

[0004] However, simply acquiring structural information indicating the degree of abnormality in the tissue structure has made it difficult for the user to appropriately determine the position of the structural information in the subject's eye.

[0005] One of the disclosed techniques aims to enable a user to appropriately determine the position of structural information in a subject's eye.

[0006] In addition to the above-mentioned objective, the present invention can also be positioned as another objective of the present invention by achieving effects that cannot be obtained by conventional technologies, which are derived from the various configurations shown in the detailed description of the invention described below. [Means for solving the problem]

[0007] One of the disclosed ophthalmic image processing devices is an ophthalmic image processing device that processes ophthalmic images that are images of tissues of a subject's eye, wherein a control unit of the ophthalmic image processing device acquires an ophthalmic image captured by an ophthalmic image capturing device, inputs the ophthalmic image to a mathematical model trained by a machine learning algorithm, thereby acquiring a probability distribution for identifying tissues in the ophthalmic image, and acquires a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality in the structure of the tissue, ophthalmology displaying an image on a display means; ophthalmology An analysis map showing the state of the tissue of the subject's eye is superimposed on an area of ​​the image where the degree of deviation is smaller than a predetermined value, and an analysis map showing the state of the tissue of the subject's eye is not superimposed on an area where the degree of deviation is larger than the predetermined value. [Effects of the Invention]

[0008] According to one of the disclosed techniques, it is possible for a user to appropriately determine the position of structural information in a subject's eye. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a system according to an embodiment of the disclosed technology. [Figure 2] FIG. 10 is a diagram illustrating an example of an operation flow of a system according to an embodiment of the disclosed technology. [Figure 3] FIG. 10 is a diagram illustrating an example of a tomographic image according to an embodiment of the disclosed technique. [Figure 4] FIG. 10 is a diagram illustrating a graph of deviation according to an embodiment of the disclosed technology. [Figure 5] FIG. 10 is a diagram illustrating the value of the degree of deviation for each combination according to an embodiment of the disclosed technology. [Figure 6] FIG. 10 is a diagram illustrating an example of a tomographic image according to an embodiment of the disclosed technique. [Figure 7] FIG. 10 is a diagram illustrating an example of a tomographic image according to an embodiment of the disclosed technique. [Figure 8]FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 9] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 10] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 11] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 12] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 13] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 14] FIG. 10 is a diagram illustrating an example of a front image according to an embodiment of the disclosed technology. [Figure 15] FIG. 10 is a diagram illustrating an example of a standard data display according to an embodiment of the disclosed technology. [Figure 16] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 17] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 18] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 19] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 20] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 21] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. [Figure 22] FIG. 10 is a diagram illustrating an example of image processing software according to an embodiment of the disclosed technology. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, exemplary embodiments for implementing the disclosed technology will be described in detail with reference to the drawings. However, the dimensions, materials, shapes, and relative positions of components described in the following embodiments are arbitrary and can be changed depending on the configuration of an apparatus to which the disclosed technology is applied or various conditions. In addition, the same reference numerals are used between drawings to indicate identical or functionally similar elements.

[0011] (Embodiment 1) In the prior art, there is a technology that inputs an ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image, and then identifies the tissue. In addition, by analyzing the intermediate values ​​used internally during tissue identification or the output values ​​after identification, it is possible to obtain the probability (likelihood) and uncertainty (deviation) of the identification result.

[0012] By quantifying the deviation, it is possible to associate the deviation with structural abnormalities. For example, when there is no abnormality in the structure of a tissue, the mathematical model is likely to accurately identify the tissue, and the obtained probability distribution is likely to be biased. On the other hand, when there is an abnormality in the structure of the tissue, the obtained probability distribution is less likely to be biased. Therefore, when the deviation value is small, the degree of structural abnormality is small, and when the deviation value is large, the degree of structural abnormality is large.

[0013] In an exemplary embodiment, the deviation is set to the inverse of the difference in scores between the first and second candidates. When the first and second candidates are closely matched (the difference in scores is small), the deviation, which is the inverse of the difference in scores, will be high. Furthermore, the method for calculating the deviation is not limited to the inverse of the difference in scores; the value of the internal parameter used for the judgment during tissue identification may be directly used, or the deviation may be set using the difference in scores between the first and second candidates during tissue identification.

[0014] The following describes a control method for an ophthalmologic image processing apparatus that displays structural information indicating the degree of structural abnormality on a display unit. The structural information indicating the degree of structural abnormality is, for example, a map or graph representing the degree of structural abnormality, and is generated using the acquired deviation. Note that the deviation is an example of the degree of structural abnormality, and values ​​such as statistical variance or likelihood of the probability distribution, such as the entropy, scatter, sum of squared deviations, standard deviation, variance, and coefficient of variation, may be used instead of the deviation as the degree of structural abnormality. A map or graph may be displayed as information indicating the statistical variance of the acquired probability distribution. Furthermore, although the deviation is assumed to be calculated during tissue identification in the ophthalmologic image in this specification, the deviation may also be calculated using the quality of the image itself (signal intensity and noise) or the difference from previous examinations. By superimposing the structural information on the image of the subject's eye, the user can appropriately determine the location of the structural information in the subject's eye.

[0015] <Device configuration> First, an outline of an ophthalmologic image processing device according to the disclosed technology will be described with reference to FIG.

[0016] One of the disclosed embodiments is an ophthalmologic image processing device including a control unit including an image acquisition unit 101, a probability distribution acquisition unit 102, a tissue identification unit 103, a deviation degree acquisition unit 104, and a display control unit 105. The image acquisition unit 101 acquires a tomographic image and notifies the probability distribution acquisition unit 102 and the display control unit 105. The probability distribution acquisition unit 102 calculates a probability distribution for tissue identification from the tomographic image and notifies the tissue identification unit 103. The tissue identification unit 103 identifies tissue information based on the probability distribution and notifies the deviation degree acquisition unit 104 and the display control unit 105. The deviation degree acquisition unit 104 acquires a deviation degree in the identified tissue information using the probability distribution for tissue identification and notifies the display control unit 105. The display control unit 105 displays the tomographic image, tissue information, and deviation degree on display means.

[0017] An exemplary ophthalmic image processing device is a personal computer (hereinafter referred to as a PC). Note that devices that can function as ophthalmic image processing devices are not limited to PCs. For example, an ophthalmic image capturing device or a server may function as an ophthalmic image processing device. When an ophthalmic image capturing device functions as an ophthalmic image processing device, the ophthalmic image capturing device can capture ophthalmic images and obtain a deviation degree from the captured ophthalmic images. Furthermore, the ophthalmic image capturing device may be configured to capture an appropriate part based on the obtained deviation degree.

[0018] The display means for displaying the structural information is, for example, a display connected to the ophthalmic image processing device. The display means may also be a device such as a display or a projector. Furthermore, the display means may be built into the ophthalmic image processing device, or built into or connected to the ophthalmic imaging device. A device different from the ophthalmic image processing device and the ophthalmic imaging device may function as the display means.

[0019] Furthermore, a tablet terminal or a mobile terminal such as a smartphone may function as the ophthalmic image processing device. Control units of multiple devices (for example, the CPU of a PC and the CPU of an ophthalmic imaging device) may cooperate to perform various processes. A cloud server capable of communicating with the ophthalmic imaging device may function as the ophthalmic image processing device. When image processing is performed on the cloud server, the acquired deviation degree or an image visually representing the deviation degree may be configured to be sent to a PC or a mobile terminal.

[0020] The mathematical model trained by the machine learning algorithm used by the probability distribution acquisition unit 102 is a mathematical model trained using a training dataset, the input side of which is data on ophthalmic images of the tissue of the subject's eye taken in the past, and the output side is data indicating the tissue in the ophthalmic images on the input side.

[0021] The control unit may input the ophthalmic image into the mathematical model to obtain a probability distribution for identifying one or more layers or boundaries included in the multiple layers and layer boundaries in the ophthalmic image. The control unit may obtain a deviation degree for one or more layers or boundaries. The control unit may obtain only the deviation degree for a layer or boundary that is likely to have a structural abnormality due to the influence of a disease.

[0022] It is not necessary to use a mathematical model trained by a machine learning algorithm in the probability distribution acquisition unit 102 and the tissue identification unit 103. For example, tissue information may be identified using a rule-based algorithm, or information in a standard database may be used to identify tissue information.

[0023] Various types of images can be used as the ophthalmic image for tissue identification. For example, the ophthalmic image may be an OCT image obtained by processing an OCT signal generated by a reference light and a measurement light reflected from the tissue of the subject's eye. The OCT image may be a two-dimensional or three-dimensional tomographic image. The tomographic image may be captured by a device other than an OCT device (e.g., a Scheimpflug camera). The ophthalmic image may also be a two-dimensional fundus image captured by illuminating the subject's eye with visible light, or an SLO image obtained by scanning the fundus with laser light using a scanning laser ophthalmoscope (SLO).

[0024] The ophthalmic image for tissue identification may be a two-dimensional en-face image (En-Face image) generated based on data from a three-dimensional tomographic image captured by an OCT device, or a two-dimensional en-face image (Motion Contrast Image) created from motion contrast data obtained by processing multiple OCT data acquired from the same position at different times.

[0025] Furthermore, the tissue to be photographed as the input ophthalmologic image can also be appropriately selected. For example, an image of the fundus, an anterior segment, an angle, or the like of the subject's eye may be used as the ophthalmologic image.

[0026] <Display organization information> 2 and 3 illustrate a method for superimposing structural information on a tomographic image.

[0027] First, in S1 of FIG. 2, the image acquisition unit 101 acquires a tomographic image 301, which is an ophthalmologic image.

[0028] In S2, the tissue identification unit 103 is applied to each A-scan An (for example, 1≦n≦128) of the acquired tomographic image 301 to identify the layer boundary L(n,m) (for example, 1≦m≦5) for An.

[0029] Furthermore, the layer boundary Lm can be determined by connecting layer boundaries L(n, m) for each m in the range of 1≦n≦128. In one disclosed embodiment, the following are identified: L1: internal limiting membrane (ILM), L2: boundary between the nerve fiber layer (NFL) and ganglion cell layer (GCL) (NFL / GCL), L3: boundary between the inner plexiform layer (IPL) and inner nuclear layer (INL) (IPL / INL), L4: boundary between the inner segment (IS) and outer segment (OS) (IS / OS), and L5: boundary between the retinal pigment epithelium (RPE) and choroid (RPE / Choroid).

[0030] In S3, the discrepancy obtaining unit 104 obtains the discrepancy D(n,m) for L(n,m). The discrepancy D(n,m) can be calculated by D(n,m) = 1 / d(n,m), where d(n,m) is the score difference between the first and second candidates when identifying the layer boundary L(n,m) for An. In this case, if the first and second candidates are close, the score difference d will be small and the discrepancy D will be large. However, the method for calculating the discrepancy D is not limited to the above-mentioned method. For example, the value of the discrepancy D may be set based on the median value during tissue identification, or the value of the discrepancy D may be set using statistical variations such as the entropy, dispersion, sum of squared deviations, standard deviation, variance, and coefficient of variation of the probability distribution, or a combination thereof.

[0031] Figure 4 is a graph showing the deviation D(n,1) at the layer boundary L1 in the tomographic image 301 of Figure 3. The horizontal axis represents the A-scan position n, and the vertical axis represents the value of the deviation D(n,1) at L1 (graphs for L2, L3, L4, and L5 are omitted). Figure 5 is a table showing the values ​​of the deviation D(n,m) in the tomographic image 301 of Figure 3, and shows that the deviation D around the anomalous structure 302 has large values ​​(D(n,1) = 4.2, D(n,2) = 3.7, D(n,3) = 3.5).

[0032] In S4, structural information indicating the degree of structural abnormality is superimposed on a tomographic image, which is an image of the eye to be examined. The structural information is, for example, a map or graph expressing the degree of structural abnormality, and is generated using the acquired deviation. As shown in FIG. 6, an area 601 surrounded by a boundary line Lm where the value of deviation D is greater than a predetermined value (hereinafter referred to as a threshold) is displayed in color, thereby emphasizing the periphery of the abnormal structure. Furthermore, brightness, saturation, transmittance, etc. may be changed according to the value of deviation D.

[0033] Furthermore, a graphic may be superimposed on the structural information of the region where the deviation D exceeds the threshold. The graphic representing the region where the deviation D exceeds the threshold may be any shape such as a circle or a rectangle.

[0034] In this embodiment, the deviation D(n,m) of the layer boundary Ln in A-scan An is focused on and highlighted, but the average value of the deviation D of the range region in multiple A-scans or multiple layer boundaries may be used for highlighting. Furthermore, if there is even a small area with a high deviation D, or if there is an area with a high deviation D of a certain level or more, a message may be displayed urging the user to check the relevant area.

[0035] Furthermore, the rendering format of layer boundaries included in regions where the deviation D exceeds a threshold value may be changed. For example, since regions with high deviations are less accurate than other regions, as shown in FIG. 7, boundary lines 701 with high deviations may be displayed in a more subdued rendering format than other regions, such as dashed lines or lighter colors. Alternatively, multiple boundary lines (e.g., first and second candidates) may be displayed to allow the user to select one, or the user may manually modify the boundary line. Furthermore, the process of detecting layer boundaries may be performed again, taking into account the user's instructions for modification.

[0036] Alternatively, instead of performing the process of extracting areas where the deviation degree D is large, a graph, map, or numerical value of the deviation degree D may be superimposed on an image of the subject's eye as structural information, thereby enabling the location of areas where the deviation degree D is large to be grasped.

[0037] Furthermore, a figure indicating the position where the structural information was acquired may be superimposed on the image of the subject's eye, and a graph or map indicating the degree of deviation D as structural information may be displayed alongside the image of the subject's eye. The figure indicating the position where the structural information was acquired may be any shape, such as a straight line, a circle, or a rectangle.

[0038] In addition, in this embodiment, the value of the degree of deviation is calculated from a tomographic image, and an area with a high degree of deviation is drawn on the same tomographic image, but the area with a high degree of deviation may be drawn on another image of the subject's eye. For example, the area with a high degree of deviation may be drawn by superimposing it on an SLO image or fundus photograph of the same location, or the results of a visual field test.

[0039] After the display of the structural information in S4 is completed, the control unit may store the ophthalmic image and the degree of deviation in the storage device. The control unit may also store an image visually displaying the degree of deviation in the storage device. Information such as structural information and analysis results may be stored in the storage device at each step, not just after the completion of S4. The storage device is a device built into or connected to the ophthalmic image processing device. The storage device may be, for example, a volatile memory or a non-volatile memory.

[0040] (Embodiment 2) One technique for generating a frontal image from multiple tomographic images is to generate a projection image by averaging the depthwise values ​​of each A-scan of multiple tomographic images and reconstructing them. Furthermore, by using the tissue identification results and performing a similar operation by averaging the values ​​between each layer on the A-scan in the depthwise direction, it is possible to generate an en-face image between any layers. Furthermore, by continuously acquiring multiple tomographic images from the same position and calculating the motion contrast, it is possible to generate an OCTA image.

[0041] In this embodiment, a method for superimposing structural information indicating the degree of structural abnormality on a front image of the subject's eye will be described. The structural information is, for example, a map or graph representing the degree of structural abnormality, and is generated using the acquired deviation D. Note that the deviation is an example of the degree of structural abnormality, and instead of the deviation, values ​​such as statistical variance or likelihood of the probability distribution, such as entropy, scatter, sum of squared deviations, standard deviation, variance, and coefficient of variation, may be used as the degree of structural abnormality. A map or graph may be displayed as information indicating the statistical variance of the acquired probability distribution. By superimposing the structural information on the front image of the subject's eye, the user can appropriately determine the position of the structural information in the subject's eye. The configuration of the apparatus and the method for acquiring the deviation D are the same as those in the first embodiment, and therefore will not be described here.

[0042] <Display organization information> FIG. 8 shows a front image 801 generated from a plurality of tomographic images including the tomographic image 301, and the tomographic image 301 corresponds to the position of the horizontal broken line on the front image 801.

[0043] Alternatively, instead of performing the process of extracting areas where the deviation degree D is large, any one of a graph, map, or numerical value of the deviation degree D may be superimposed on an image of the eye to be examined, thereby enabling the position of areas where the deviation degree D is large to be grasped. In Fig. 9, a deviation degree color map corresponding to the value of the deviation degree at each coordinate of the front image 801 is drawn on the entire surface of the front image 801 (drawing area 901).

[0044] The deviation of each coordinate may be the average deviation of the A-scans that make up the coordinate, or the deviation value between any layers. Here, the deviation color map can visually depict the deviation by coloring areas with high deviation (around the anomalous structure 302) in dark colors and areas with low deviation (areas far from the anomalous structure 302) in light colors.

[0045] Furthermore, the method of expressing the deviation color map is not limited to color density; lightness, saturation, transmittance, etc. may be changed according to the deviation. Furthermore, the color density, lightness, saturation, transmittance, etc. of the deviation color map may be configured to be changeable in response to a user instruction. User instructions may be input using operations such as double-clicking, dragging, or scrolling. Furthermore, a scroll bar or the like may be displayed to change the color density, lightness, saturation, transmittance, etc.

[0046] 10, the deviation color map may be drawn only in the region (drawing region 1001) where the deviation D is greater than a threshold value (e.g., 3.0), thereby emphasizing the high deviation portion. In this case, the drawing region 1001 may be a region where there is at least one layer where the deviation D value is greater than the threshold value, or may be a region where the average value of the deviation D of each layer is greater than a threshold value (e.g., 2.0).

[0047] Alternatively, the rendering region 1001 may be set using only the deviation value of a specific layer. For example, in this embodiment, in the A-scan An of the tomographic image 301, the deviation values ​​of L1, L2, and L3 exceed the threshold value of 3.0, but L4 and L5 do not exceed the threshold value of 3.0. Therefore, when displaying the deviation based on L1, L2, and L3, the rendering region 1001 including the abnormal structure 302 is rendered, but when displaying the deviation based on L4 and L5, the rendering region 1001 is not rendered. In this way, the rendering region 1001 may be switched for each layer.

[0048] Furthermore, the region where the deviation D exceeds the threshold may be displayed by being surrounded by a graphic. The graphic representing the region where the deviation D exceeds the threshold may be any shape such as a circle or a rectangle.

[0049] Furthermore, a figure indicating the position where the deviation degree D was obtained may be superimposed on the image of the subject's eye, and a graph or map indicating the deviation degree D may be displayed alongside the image of the subject's eye as structural information. The figure indicating the position where the deviation degree D was obtained may be any shape, such as a straight line, a circle, or a rectangle.

[0050] In addition, in this embodiment, the value of the deviation degree is calculated from a tomographic image, and the drawing area 1001 is drawn on the front image 801 generated using the same tomographic image, but the drawing area 1001 may also be drawn on another image of the subject's eye. For example, the drawing area 1001 may be drawn by superimposing it on an SLO image or fundus photograph taken of the same location, or the results of a visual field test.

[0051] In addition, in this embodiment, the drawing area 1001 is specified using the value of the deviation degree calculated from the tomographic image, but by training a mathematical model in advance using a large number of frontal images, the frontal image 801 can be input into the mathematical model to calculate the value of the deviation degree in the frontal image and then the drawing area 1001 can be specified.

[0052] (Embodiment 3) In this embodiment, a method for displaying the deviation degree in combination with other display items will be described. By displaying an analysis map showing the structural state of the subject's eye in combination with structural information showing the degree of structural abnormality, the user can easily determine the location of the structural information in the subject's eye, taking into account both the structural state and the degree of structural abnormality. The structural information is, for example, a map, graph, or numerical value representing the degree of structural abnormality, and is generated using the acquired deviation degree D. Note that the deviation degree is an example of the degree of structural abnormality, and values ​​such as statistical variations and likelihoods of the probability distribution, such as the entropy, scatter, sum of squared deviations, standard deviation, variance, and coefficient of variation, may be used as the degree of structural abnormality instead of the deviation degree. Maps, graphs, and numerical values ​​may be displayed as information showing the statistical variations of the acquired probability distribution. The configuration of the device and the method for acquiring the deviation degree D are the same as those in the first embodiment, and therefore will not be described here.

[0053] It is possible to calculate the thickness of each layer from a tomographic image and display it as a layer thickness color map, which is one of the analysis maps, on the frontal image. The generation of a layer thickness color map is premised on accurate detection of layer boundaries, and if there is an error in the detected layer boundaries, an accurate layer thickness color map cannot be displayed. Therefore, by combining and displaying the layer thickness color map with structural information, it is possible to display a highly accurate layer thickness color map.

[0054] 11 shows an example of a layer thickness color map, which is one of the analysis maps, combined with structural information and drawn on a front image 801. In one embodiment, a layer thickness color map (blackened area) is drawn in a drawing area 1001 excluding an area 1002 on the front image 801 where the deviation exceeds a threshold value (e.g., 3.0). This makes it possible to draw a highly accurate layer thickness color map excluding the abnormal structure 302.

[0055] Furthermore, the high deviation region 1002 may be depicted in a different color or color map from the layer thickness color map (drawing region 1001), or as shown in FIG. 12, the high deviation region 1002 may be used as the drawing region 1001 to draw the layer thickness color map. The color density, brightness, saturation, transmittance, and the like of the layer thickness color map may be changed in response to a user instruction. The user's instruction may be input using an operation such as double-clicking, dragging, or scrolling. A scroll bar or the like may be displayed to change the color density, brightness, saturation, transmittance, and the like.

[0056] Furthermore, the analysis map is not limited to a layer thickness map. The analysis map may be a map showing the condition of the subject's eye, such as the results of a visual field test or blood vessel density. As one of the analysis maps, a map of deviation indicating the degree of structural abnormality may be superimposed on an image of the subject's eye. Furthermore, an image of the subject's eye and a graph of deviation may be displayed side by side, and a straight line indicating the position where the graph of deviation was obtained may be superimposed on the structural information (on the map of deviation). The shape displayed on the structural information may be any shape, such as a rectangle or an arrow, in addition to a straight line.

[0057] Next, a method for displaying a grid display in combination with the degree of deviation will be described. Generally, a technology is used in which a grid, which is a graphic divided into multiple regions, is displayed on a frontal image, and values ​​such as layer thickness and blood vessel density within each region of the grid are measured and displayed. Measurement of layer thickness, blood vessel density, etc. is premised on accurate detection of layer boundaries, and if the detected layer boundaries contain errors, accurate values ​​such as layer thickness and blood vessel density cannot be displayed. Therefore, by displaying a grid display in combination with the degree of deviation, it is possible to perform a grid display that makes it easy to determine the accuracy of values ​​such as layer thickness and blood vessel density.

[0058] 13 is a diagram showing an ETDRS grid 1301, which is one of the figures divided into multiple regions, superimposed on the front image 801. In Fig. 13(a), the deviation grid 1301(a) displays the average deviation value within each region, and grids 1302 whose average deviation value exceeds a threshold value (e.g., 3.0) are displayed in color. This makes it possible to visually recognize that the colored grid 1302 has a high deviation value and therefore an abnormal structure exists.

[0059] 13(b), the average layer thickness in each region is displayed in the layer thickness grid 1301(b), and each grid is colored based on the layer thickness value. Also, by switching between the deviation grid 1301(a) and the layer thickness grid 1301(b), the relationship between the deviation and the layer thickness can be easily visualized.

[0060] Furthermore, the layer thickness grid 1301(c) in FIG. 13(c) is displayed in color based on the value of the deviation grid 1301(a) relative to the layer thickness grid 1301(b). This makes it easier to visually recognize the relationship between deviation and layer thickness. Furthermore, as shown in the layer thickness-deviation grid 1301(d) in FIG. 13, the deviation and layer thickness may be displayed together in each grid. In FIG. 13(d), the layer thickness-deviation grid 1301(d) is displayed in color based on the deviation, but it may also be displayed in color based on the layer thickness, as in the layer thickness grid 1301(b).

[0061] 13(e), the layer thickness / deviation grid 1301(e) is displayed by simultaneously coloring both the deviation grid 1301(a) and the layer thickness grid 1301(b). Specifically, the colored portion of the deviation grid 1301(a) is painted black, while the layer thickness grid 1301(b) is colored. These display methods are merely examples, and the colors of the grids may be mixed, or a display method other than color, such as a combination of patterns or symbols, may also be used.

[0062] Next, a method for displaying structural information in comparison with past examinations will be described. For follow-up observations, etc., there are cases where observations are made in comparison with past examinations. In such cases, it is possible to visualize the progress by displaying the difference in the degree of deviation as structural information based on the degree of deviation stored in a storage device.

[0063] FIG. 14(a) shows a front image 801 captured of a certain subject's eye, and FIG. 14(b) shows a front image 801' captured of the same subject's eye on a different day. The abnormal structure 302' enlarges, and accordingly, the region 1401 (1401') where the deviation exceeds the threshold also enlarges. The progress can be visualized by drawing a region 1402 on the front image 801', which is the difference obtained by subtracting the region 1401 before enlargement from the enlarged region 1401'. While this example shows an example in which the abnormal structure 302' enlarges, the difference when the region 1401' shrinks due to the shrinkage of the abnormal structure 302' can also be similarly expressed. It is preferable to use different methods of expression, such as red when the region 1401' enlarges and blue when it shrinks. Alternatively, the region 1401 may simply be drawn on the front image 801, and the enlarged region 1401' may be drawn on the front image 801', and displayed side by side.

[0064] In addition, in this embodiment, the deviation value is calculated from a tomographic image, and a grid or comparison information with a previous examination is plotted on a front image generated using the same tomographic image, but the grid or comparison information with a previous examination may also be plotted on another image of the subject's eye. For example, the grid or comparison information with a previous examination may be plotted by superimposing it on an SLO image or fundus photograph of the same location, or the results of a visual field test.

[0065] Next, a method for displaying a combination of standard data and deviation will be described. Standard data is data that indicates the average for normal eyes for a combination of age, race, etc., and deviations from the standard data can be considered to be abnormal. Figure 15 shows an example of a method for displaying a combination of standard data and deviation. Standard data graph 1501 shows the A-scan position on the horizontal axis and the layer thickness (μm) for each A-scan position on the vertical axis. Standard data (dashed line) 1502 indicates the average range for normal eyes indicated by the standard data. Test eye data (solid line) 1503 is the actual value of the test eye, and any point where test eye data 1503 exceeds standard data 1502 can be considered to deviate from the normal value.

[0066] Here, in the standard data graph 1501, an area 1504 that "deviates from the normal value and has a low degree of deviation" and an area 1505 that "deviates from the normal value and has a high degree of deviation" are displayed in different ways (for example, 1504 is blue and 1505 is red). Area 1504 has a low degree of deviation, which indicates that it deviates from the normal value but is not an abnormal structure, while area 1505 has a high degree of deviation, which indicates that it deviates from the normal value due to an abnormal structure. As described above, it is possible to express the degree of deviation as auxiliary information when using standard data.

[0067] In addition to the contents described in this embodiment, the deviation may be displayed in combination with the panoramic synthesis result, the automatic diagnosis result, etc. Furthermore, the next time an image is taken, the area, angle of view, focus, etc. may be automatically set so that an area with a high deviation in the previous examination is focused on.

[0068] (Embodiment 4) In this embodiment, a specific method for displaying structural information and an image of a subject's eye will be described. The structural information is, for example, a map, graph, or numerical value representing the degree of structural abnormality, and is generated using the acquired deviation D. Note that the deviation is an example of the degree of structural abnormality, and values ​​such as statistical variations or likelihoods of the probability distribution, such as the entropy, scatter, sum of squared deviations, standard deviation, variance, and coefficient of variation, may be used as the degree of structural abnormality instead of the deviation. Maps, graphs, and numerical values ​​may be displayed as information indicating the statistical variations of the acquired probability distribution. Furthermore, the method for displaying structural information described in this embodiment is an example of the disclosed technology, and can be implemented by combining embodiments 1 to 3. The configuration of the apparatus and the method for acquiring the deviation D are the same as those in embodiment 1, and therefore will not be described here.

[0069] 16 shows a report screen 1600 of the image processing software. The report screen 1600 displays a front image 1601 and a tomographic image 1602. The tomographic image 1602 is a tomographic image at a position corresponding to a line 1603 on the front image 1601. A configuration may be adopted in which the tomographic image including the area with the highest degree of deviation from among the acquired tomographic images is preferentially displayed as the tomographic image 1602.

[0070] The user may input an instruction to move the position of the line 1603, thereby switching the tomographic image 1602 accordingly. The user may input an instruction by double-clicking, dragging, scrolling, or other operations. Alternatively, a configuration may be adopted in which radio buttons for selecting a line or a scroll bar for moving the line are displayed.

[0071] Furthermore, the display of the image of the subject's eye is not limited to the display of a tomographic image and a frontal image. For example, the structural information may be displayed by superimposing it on an SLO image or a fundus photograph of the same area, or the results of a visual field test.

[0072] In addition, by switching the front image selection field 1604, the image displayed in the front image 1601 can be switched. In particular, when selecting a front image generated using a tomographic image, such as an En-Face image or an OCTA image, in the front image selection field 1604, the parameters used for generation can be changed. For example, by specifying an arbitrary layer boundary in the layer boundary specification field 1606, a front image between the specified layers can be displayed. In addition, the range selection field 1605 includes Superficial ("internal limiting membrane (ILM)" and "internal plexiform layer (IPL)") and Deep ("internal nuclear layer (INL)" and "external limiting membrane (ELM)"), and by selecting one of these, an appropriate value is automatically input to the layer boundary specification field 1606.

[0073] The display switching unit 1607 has buttons for switching the layer thickness map and layer thickness grid ON / OFF and for displaying / hiding the degree of deviation, and switches between a front image 1601 and a tomographic image 1602 depending on the state of the button. For example, when the degree of deviation in the display switching unit 1607 is set to "display" as shown in Fig. 16, an area 1608 with a high degree of deviation in the tomographic image 1602 is displayed in color (corresponding to the tomographic image 301 in Fig. 6), and a corresponding area 1609 with a high degree of deviation in the front image 1601 is displayed in color (corresponding to the front image 801 in Fig. 10).

[0074] Furthermore, the configuration may be such that the color density, brightness, saturation, transparency, etc. of the color used to draw the high deviation area can be changed in response to a user instruction. The user's instruction may be input by double-clicking, dragging, scrolling, etc. Furthermore, the configuration may be such that a scroll bar or the like is displayed to change the color density, brightness, saturation, transparency, etc.

[0075] 17 shows an example in which the layer thickness map of the display switching unit 1607 is ON and the deviation degree is "display." In this case, the layer thickness map is displayed in an area excluding an area 1701 with a high deviation degree (corresponding to the drawing area 1001 in FIG. 11). On the other hand, when the layer thickness map is ON and the deviation degree is "not displayed," the layer thickness map is overlaid on the entire front image 1601 (corresponding to the drawing area 901 in FIG. 9).

[0076] 18 shows an example in which the layer thickness grid of the display switching unit 1607 is ON and the deviation is "displayed." In this case, a layer thickness / deviation grid 1801 (corresponding to the layer thickness / deviation grid 1301(d) in FIG. 13(d)) is displayed in which grid portions with high deviation are colored. On the other hand, when the layer thickness grid is ON and the deviation is "not displayed," a layer thickness grid (corresponding to the layer thickness grid 1301(b) in FIG. 13(b)) is displayed in which each grid is colored based on the layer thickness.

[0077] Furthermore, in each display, the display method based on the degree of deviation may be changed depending on the values ​​in the range selection section 1605 and the layer boundary designation section 1606. For example, the average value of the degrees of deviation of each A-scan in the tomographic image 1602 may be used, or the average value of the degrees of deviation between only the layers designated in the layer boundary designation section 1606 may be used, or the maximum or minimum value of the degree of deviation within the range may be used.

[0078] Furthermore, since an area with a high degree of deviation is likely to be an abnormal structure, as shown in FIG. 19, when an area 1609 with a high degree of deviation on the front image 1601 is clicked, all or a plurality of tomographic images corresponding to that area (group of tomographic images 1901) may be displayed side by side on the screen. If the group of tomographic images 1901 does not fit on the screen, they may be displayed in a separate dialog, or the group of tomographic images 1901 may be switched for display. At this time, the layer boundaries of each tomographic image in the group of tomographic images 1901 may be manually adjusted. The group of tomographic images 1901 may also be saved to a file or printed all at once. The layer boundaries of each tomographic image may also be manually adjusted.

[0079] Furthermore, a partial region image of the ophthalmologic image in a region with a high degree of deviation may be displayed, or a process may be performed to enlarge the ophthalmologic image in a region with a high degree of deviation and display the enlarged image.

[0080] 20, confirmation information 2001 indicating whether or not a high deviation area 1609 has been confirmed may be added. In addition to a confirmed checkbox, the confirmation information 2001 may automatically input the confirmation date and the person who confirmed it, and may also include a remarks field for freely entering the diagnosis results, countermeasures, etc. In addition to the confirmation information, an instruction to re-photograph the high deviation area may be displayed.

[0081] Furthermore, the high deviation area 1609 and confirmation information 2001 may be collected and re-learned using a machine learning algorithm, and used for automatic determination of high deviation areas, etc. Furthermore, if an attempt is made to transition to another screen while there are still items that have not been confirmed, a confirmation dialog 2101 as shown in FIG.

[0082] Furthermore, inspections with high deviations may be selected and displayed during screening. Fig. 22 shows an inspection selection screen 2200 for screening of image processing software. Each row in an inspection list 2201 displays inspection information, along with the deviation value for each inspection. Here, rows 2202 of inspections with deviation D values ​​exceeding a threshold value (e.g., 3.0) are displayed in color, making it possible to effectively discover inspections with high deviations (including abnormal structures) during screening.

[0083] Furthermore, if a subject has undergone multiple consecutive examinations, the image with the highest degree of deviation may be displayed preferentially, or the images may be sorted and displayed in order of degree of deviation. Furthermore, for an examination with a high degree of deviation, the subject may be asked to re-examine the subject using the same settings, or may be asked to re-examine using settings changed to high-resolution imaging. Furthermore, a high-resolution image may be generated by capturing multiple images of the same location and averaging the captured images.

[0084] Furthermore, during a follow-up examination, a region that showed a high degree of deviation in the previous examination may be examined with priority. The degree of deviation obtained in the follow-up examination may be stored in a storage device, and multiple ophthalmologic images of the same eye taken at different times may be displayed along with the degree of deviation for each image.

[0085] (Embodiment 5) In this embodiment, modifications of the first to fourth embodiments will be described.

[0086] In the first to fourth embodiments, a method for displaying structural information based on the deviation has been described, but the present invention is not limited to this. For example, instead of the deviation, display may be based on another index such as likelihood or image quality (signal strength or noise). For example, in FIG. 7 of the first embodiment, a portion with low likelihood or a portion with low image quality may be displayed by a dashed line 701. Similarly, in the second and third embodiments, another index may be used instead of the deviation.

[0087] Furthermore, in the fourth embodiment, it may be possible to display another index such as image quality instead of the deviation in the display switching unit 1607 in Fig. 16 . In this case, when the image quality is set to "display," an area 1609 where the signal intensity is low or there is a lot of noise is drawn in color. It may also be possible to set the deviation and another index such as image quality in parallel. In this case, when the deviation is set to "display" and the image quality is set to "display," the area that is the sum of the area colored by the deviation and the area colored by the image quality may be colored as 1609, or the area that is the product may be colored as 1609.

[0088] Furthermore, the deviation value may be used to calculate another value. For example, the deviation may be used to evaluate image quality. A QI (Quality Index) value is generally set as a guideline for image quality, and the QI value may be calculated based on the deviation. Furthermore, a QI value set based on another value may be corrected based on the deviation.

[0089] In the above-described first to fifth embodiments, an ophthalmological image processing device has been described as an example of an embodiment of the disclosed technology, but an ophthalmological image processing program that executes the same functions may also be used. The program that executes the above-described functions is supplied to a system or device via a network or various storage media. The computer of the system or device can then read and execute the program.

Claims

1. An ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of a subject's eye, The control unit of the ophthalmologic image processing device Acquire an ophthalmic image captured by an ophthalmic image capturing device; inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image; acquiring a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality in the structure of the tissue; displaying the ophthalmologic image on a display means; An ophthalmologic image processing device that superimposes an analysis map showing the state of the tissue of the test eye in an area of ​​the ophthalmologic image where the degree of deviation is smaller than a predetermined value, and does not superimpose the analysis map showing the state of the tissue of the test eye in an area where the degree of deviation is larger than the predetermined value.

2. The ophthalmologic image processing apparatus according to claim 1 , wherein the control unit of the ophthalmologic image processing apparatus colors an area of ​​the ophthalmologic image where the degree of deviation is greater than a predetermined value, and causes the display unit to display the ophthalmologic image.

3. 3 . The ophthalmologic image processing apparatus according to claim 1 , wherein the control unit of the ophthalmologic image processing apparatus displays a graphic representing an area where the degree of deviation is greater than a predetermined value, superimposed on the structural information of the subject's eye.

4. the ophthalmologic image is a tomographic image of the subject's eye, 4. The ophthalmological image processing device according to claim 1, wherein a control unit of the ophthalmological image processing device displays, in different drawing formats, the boundary lines of the tissues included in the area where the deviation degree is greater than a predetermined value and the boundary lines of the tissues included in the area where the deviation degree is smaller than the predetermined value in the tomographic image of the test eye.

5. the ophthalmologic image is a front image of the subject's eye, 4. The ophthalmological image processing device according to claim 1, wherein a control unit of the ophthalmological image processing device switches between superimposing the front image and an analysis map showing the state of the tissue of the test eye and superimposing the front image and structural information showing the degree of abnormality in the structure of the tissue in response to a user instruction.

6. the ophthalmologic image is a front image of the subject's eye, a control unit of the ophthalmologic image processing device that causes a display unit to display the front image; 4. An ophthalmologic image processing device according to claim 1, wherein an analysis map showing the state of the tissue of the test eye is superimposed on an area of ​​the front image where the degree of deviation is smaller than a predetermined value, and an analysis map showing the state of the tissue of the test eye is not superimposed on an area where the degree of deviation is larger than the predetermined value.

7. The ophthalmological image processing device according to any one of claims 1 to 6, wherein a control unit of the ophthalmological image processing device superimposes the difference in the degree of deviation obtained from a plurality of ophthalmological images taken at different times on the ophthalmological image as structural information.

8. The ophthalmological image processing device according to any one of claims 1 to 7, wherein a control unit of the ophthalmological image processing device superimposes a graphic indicating a position where structural information indicating the degree of abnormality of the structure was acquired on the ophthalmological image.

9. An ophthalmic image processing device that processes an ophthalmic image that is an image of tissue of a subject's eye, The control unit of the ophthalmologic image processing device Acquire an ophthalmic image captured by an ophthalmic image capturing device; inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image; acquiring a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality in the structure of the tissue; acquiring standard data corresponding to the subject's eye; displaying the ophthalmologic image on a display means; an ophthalmologic image processing device that superimposes on the ophthalmologic image a result of comparing the tissue structure of the standard data with the tissue structure of the subject's eye and structural information indicating the degree of abnormality of the structure.

10. the figure indicating the position where the structural information indicating the degree of structural abnormality was acquired is a straight line; 10. The ophthalmologic image processing apparatus according to claim 8, wherein the control unit of the ophthalmologic image processing apparatus causes the display means to display the structural information as a graph showing the degree of abnormality in the structure of the tissue.

11. the figure indicating the position where the structural information indicating the degree of structural abnormality was acquired is a rectangle; 10. The ophthalmologic image processing apparatus according to claim 8, wherein the control unit of the ophthalmologic image processing apparatus causes the display means to display the structural information as a map showing the degree of abnormality in the structure of the tissue.

12. a figure indicating a position where structural information indicating the degree of structural abnormality was acquired is divided into a plurality of regions; 10. The ophthalmologic image processing apparatus according to claim 8, wherein the control unit of the ophthalmologic image processing apparatus causes the display means to display structural information indicating the degree of abnormality in the structure of the tissue for each of the plurality of regions.

13. The ophthalmologic image processing device according to claim 12 , wherein the control unit of the ophthalmologic image processing device changes the colors in the plurality of regions depending on the degree of abnormality in the tissue structure.

14. The ophthalmologic image processing device according to claim 12 or 13, wherein the control unit of the ophthalmologic image processing device displays structural information in the plurality of regions as numerical values ​​indicating the degree of abnormality of the structure of the tissue.

15. The ophthalmologic image processing device according to claim 1 , wherein the structural information indicating the degree of abnormality of the tissue structure displayed on the display means is a map with different colors depending on the degree of abnormality of the tissue structure.

16. The ophthalmologic image processing apparatus according to claim 15 , wherein at least one of brightness, saturation, and transmittance of the map of structural information indicating the degree of abnormality in the tissue structure is changed in response to a user instruction.

17. The ophthalmologic image processing apparatus according to claim 1 , wherein the degree of deviation includes at least one of entropy, sum of squared deviation, variance, standard deviation, and coefficient of variation of the acquired probability distribution.

18. the ophthalmic image is a two-dimensional or three-dimensional tomographic image of the tissue; The control unit inputting the ophthalmic image into the mathematical model to obtain the probability distribution for identifying one or more layers or boundaries included in a plurality of layers and layer boundaries in the ophthalmic image; The ophthalmologic image processing apparatus according to claim 1 , wherein the degree of deviation is acquired for the one or more layers or boundaries.

19. 19. The ophthalmologic image processing device according to claim 1, wherein the control unit executes at least one of a process of outputting an instruction to the ophthalmologic image capturing device to capture an image of an area of ​​the tissue where the degree of deviation is greater than a predetermined value, and a process of displaying a tomographic image or an enlarged image of the area where the degree of deviation is greater than the predetermined value on a display means.

20. 20. The ophthalmological image processing device according to claim 1, wherein the control unit stores the acquired deviation degree in a storage device and causes a display unit to display a plurality of deviation degrees for each of a plurality of ophthalmological images taken of the tissue of the same test eye at different times.

21. The ophthalmologic image processing apparatus according to claim 1 , wherein the control unit generates information for evaluating image quality of the ophthalmologic image based on the degree of deviation acquired for the ophthalmologic image.

22. 4. The ophthalmological image processing device according to claim 1, wherein the ophthalmological image to be superimposed is at least one of an OCT image obtained by processing an OCT signal generated by a reference light and a reflected light of a measurement light irradiated onto the tissue of the test eye, a fundus image obtained by illuminating the test eye with visible light, and an SLO image obtained by scanning the fundus with a laser light.

23. An OCT apparatus that captures an ophthalmic image, which is an image of tissue of a subject's eye, by processing an OCT signal generated by a reference light and a measurement light irradiated onto the tissue of the subject's eye and reflected light from the measurement light, The control unit of the OCT device inputting the captured ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image; acquiring a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified as structural information indicating a degree of abnormality in the structure of the tissue; displaying the ophthalmologic image on a display means; An OCT device that superimposes an analysis map showing the state of the tissue of the test eye in areas of the ophthalmic image where the degree of deviation is smaller than a predetermined value, and does not superimpose an analysis map showing the state of the tissue of the test eye in areas where the degree of deviation is larger than the predetermined value.

24. A control method for an ophthalmic image processing device that processes an ophthalmic image, which is an image of tissue of a subject's eye, comprising: acquiring an ophthalmic image captured by an ophthalmic image capturing device; inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image; acquiring, as structural information indicating a degree of abnormality of the structure of the tissue, a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified; displaying the ophthalmologic image on a display means, superimposing an analysis map showing the state of tissue of the subject's eye on an area of ​​the ophthalmologic image where the degree of deviation is smaller than a predetermined value, and not superimposing the analysis map showing the state of tissue of the subject's eye on an area of ​​the ophthalmologic image where the degree of deviation is larger than the predetermined value; A control method for an ophthalmologic image processing device including:

25. A control method for an ophthalmic image processing device that processes an ophthalmic image, which is an image of tissue of a subject's eye, comprising: acquiring an ophthalmic image captured by an ophthalmic image capturing device; inputting the ophthalmic image into a mathematical model trained by a machine learning algorithm to obtain a probability distribution for identifying tissues in the ophthalmic image; acquiring, as structural information indicating a degree of abnormality of the structure of the tissue, a degree of deviation of the acquired probability distribution from the probability distribution when the tissue is accurately identified; acquiring standard data corresponding to the subject's eye; a step of displaying the ophthalmologic image on a display means, and superimposing a result of comparing the tissue structure of the standard data with the tissue structure of the subject's eye and structural information indicating the degree of abnormality of the structure on the ophthalmologic image; A control method for an ophthalmologic image processing device including:

26. A program that causes a computer to execute the control method for an ophthalmologic image processing apparatus according to claim 24 or 25.

Images