Ocular fundus image processing device and ocular fundus image processing program

Abstract

To provide an ocular fundus image processing device and an ocular fundus image processing program capable of automatically detecting presence or absence of an artery vein crossing phenomenon with a high degree of precision by processing an ocular fundus image.SOLUTION: A control unit of an ocular fundus image processing device detects at least one of the artery and the vein in a crossing part where the artery and the vein cross in an ocular fundus image by processing a blood vessel image. The control unit acquires vein pixel information, which is pixel information on a region separated from the crossing part of the veins shown in the ocular fundus image. The control unit determines whether an artery vein crossing phenomenon is caused in the crossing part by comparing pixel information in an attention region of the ocular fundus image with the vein pixel information, with a region spreading along the vein from a contour line of the artery of the crossing part of the ocular fundus image as the attention region.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a fundus image processing device that processes a fundus image of a subject's eye, and a fundus image processing program executed in the fundus image processing device. [Background technology]

[0002] Observing the fundus makes it possible to grasp the state of blood vessels in a living body non-invasively. The retina at the fundus is very thin, so at the crossing point where an artery and a vein cross within the retina, the outer layers of the artery and the vein are shared. When arteriosclerosis occurs, the artery crushes the vein at the crossing point, and as a result, an arteriovenous crossing phenomenon occurs in which the state of the vein at the crossing point observed at the fundus changes compared to when arteriosclerosis does not occur. Therefore, the presence or absence of the arteriovenous crossing phenomenon at the fundus is a useful guideline for diagnosing arteriosclerosis in a subject. Conventionally, it has been common to determine that an arteriovenous crossing phenomenon has occurred when the vein at the crossing point observed from the front side appears to be narrowed.

[0003] A technique has also been proposed that aims to automatically detect venous stenosis at the crossing of a fundus image by processing the fundus image. For example, a detection device described in Patent Document 1 calculates the ratio of the outer diameter of the vein at a site away from the crossing of an artery and a vein to the outer diameter at a site close to the crossing as the venous stenosis at the crossing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-253485 A Summary of the Invention

[0005] As a result of sincere research by the inventors of the present application, it has been newly discovered that the state of the vein at the crossing point observed when the arteriovenous crossing phenomenon occurs is not limited to a stenotic state. For example, when the vein at the crossing point is crushed in the depth direction of the fundus due to arteriosclerosis, the vein does not necessarily look stenotic even when observed from the front side. Therefore, it has been difficult to automatically detect the presence or absence of the arteriovenous crossing phenomenon with high accuracy using conventional methods.

[0006] A typical object of the present disclosure is to provide a fundus image processing device and a fundus image processing program capable of automatically detecting the presence or absence of an arteriovenous crossing phenomenon with high accuracy by processing a fundus image. [Means for solving the problem]

[0007] A fundus image processing device provided by a typical embodiment of the present disclosure is a fundus image processing device that processes a fundus image of a test eye, and a control unit of the fundus image processing device executes a fundus image acquisition step of acquiring a fundus image including blood vessels of the fundus of the test eye photographed by a fundus image photographing device, a vascular image acquisition step of acquiring vascular images of an artery and a vein included in the acquired fundus image, a crossing detection step of detecting at least one of the crossings where an artery and a vein cross in the fundus image by processing the vascular image, a vein pixel information acquisition step of acquiring vein pixel information which is pixel information of a portion of the vein appearing in the fundus image that is spaced from the crossing, and a crossing phenomenon determination step of determining whether or not an arteriovenous crossing phenomenon occurs at the crossing by setting an area of ​​the crossing in the fundus image that extends from the contour of the artery along the vein as an attention area, and comparing pixel information within the attention area in the fundus image with the vein pixel information.

[0008] A fundus image processing program provided by a typical embodiment of the present disclosure is a fundus image processing program executed by a fundus image processing device that processes a fundus image of a test eye, and the fundus image processing program is executed by a control unit of the fundus image processing device to cause the fundus image processing device to execute the following steps: a fundus image acquisition step of acquiring a fundus image including blood vessels at the fundus of the test eye, captured by a fundus image capturing device; a vascular image acquisition step of acquiring vascular images of an artery and a vein included in the acquired fundus image; a crossing detection step of detecting at least one of the crossings where an artery and a vein cross in the fundus image by processing the vascular image; a vein pixel information acquisition step of acquiring vein pixel information, which is pixel information of a portion of the vein captured in the fundus image that is spaced from the crossing; and a crossing phenomenon determination step of determining whether or not an arteriovenous crossing phenomenon occurs at the crossing by setting an area of ​​the crossing in the fundus image that extends from the contour of the artery along the vein as an attention area, and comparing pixel information within the attention area in the fundus image with the vein pixel information.

[0009] According to the fundus image processing device and fundus image processing program of the present disclosure, the presence or absence of arteriovenous crossing is automatically detected with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a fundus image processing device 1 and a fundus image photographing device 11. FIG. [Diagram 2] 1 is a diagram showing an example of a fundus image 30 and blood vessel images 40A and 40B showing blood vessels included in the fundus image 30. FIG. [Diagram 3] 4 is a flowchart of fundus image processing executed by the fundus image processing device 1. [Figure 4] This is a diagram comparing a blood vessel image 40 with a thinned image 41 obtained by performing thinning processing on the blood vessel image 40. [Diagram 5] 4 is an explanatory diagram for explaining a method for detecting a branching point of a blood vessel in the present embodiment. FIG. [Figure 6]10 is an explanatory diagram for explaining a method for identifying blood vessel groups 53S, 53T, and 53U in this embodiment. FIG. [Figure 7] 5 is an explanatory diagram for explaining a method for detecting a crossover portion 70 in the present embodiment. FIG. [Figure 8] 1 is a diagram showing an example of a vein pixel information acquisition position 81 on a blood vessel image 40. FIG. [Figure 9] 1 is a diagram showing an example of a vein pixel information acquisition position 81 and an external pixel information acquisition position 82 on a fundus image 30. FIG. [Figure 10] 1 is a diagram showing an example of an external pixel information acquisition position 82 on a blood vessel image 40. FIG. [Figure 11] FIG. 13 is an explanatory diagram for explaining an example of a state in which vein end positions 521P, 522P, a reference line BL, and a search line SL are set. [Figure 12] FIG. 12 is an explanatory diagram for explaining a state in which the search line SL has been moved from the position shown in FIG. 11 to a vein end position 521P along the reference straight line BL. [Figure 13] FIG. 13 is an explanatory diagram for explaining a state in which the search line SL is moved along a thinned vein 52B from the position shown in FIG. 12. [Figure 14] 1 is a table showing validation results of the algorithm of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <Summary> The control unit of the fundus image processing device exemplified in the present disclosure executes a fundus image acquisition step, a blood vessel image acquisition step, a crossing portion detection step, a vein pixel information acquisition step, and a crossing phenomenon determination step. In the fundus image acquisition step, the control unit acquires a fundus image including blood vessels of the fundus of the subject eye photographed by the fundus image photographing device. In the blood vessel image acquisition step, the control unit acquires blood vessel images of an artery and a vein included in the fundus image. In the crossing portion detection step, the control unit processes the blood vessel image to detect at least one of the crossing portions where an artery and a vein cross in the fundus image. In the vein pixel information acquisition step, the control unit acquires vein pixel information, which is pixel information of a portion of the vein appearing in the fundus image that is separated from the crossing portion. In the crossing phenomenon determination step, the control unit determines whether or not an arteriovenous crossing phenomenon occurs at the crossing portion by determining, as a region of interest, a region of the crossing portion of the fundus image that extends from the contour of the artery along the vein, and compares pixel information in the region of interest in the fundus image with the vein pixel information.

[0012] When arteriosclerosis occurs, the veins at the crossings may be crushed in the depth direction of the fundus, resulting in a decrease in thickness. In this case, the amount of blood in the veins at the crossings is reduced in the depth direction compared to veins that are not crushed in the depth direction. Therefore, when the veins at the crossings are crushed in the depth direction, they tend to appear lighter in color than veins that are not crushed in the depth direction, regardless of whether they appear to be constricted or not. The inventors of the present invention have discovered the technology of the present disclosure based on the above new findings.

[0013] According to the technology of the present disclosure, the region of the intersection of the fundus image that spreads from the contour of the artery along the vein is set as the region of interest. If no arteriovenous intersection occurs, the vein crosses the artery at the intersection with a substantially constant thickness. Therefore, the pixel information in the region of interest in the fundus image is easily approximated to the vein pixel information of a portion distant from the intersection. On the other hand, when the color of the vein near the intersection is lighter due to the effect of the arteriovenous intersection, and when the vein is narrowed, the pixel information in the region of interest in the fundus image is difficult to approximate to the vein pixel information. Therefore, by comparing the pixel information in the region of interest in the fundus image with the vein pixel information, it becomes easy to appropriately detect the occurrence of the arteriovenous intersection not only when the vein is narrowed, but also when the color of the vein near the intersection is lighter. Therefore, the presence or absence of the arteriovenous intersection can be easily detected automatically with high accuracy.

[0014] In addition, the arteriovenous crossing phenomenon is likely to occur near the crossing. In the present disclosure, the vein pixel information compared with the pixel information in the region of interest is pixel information of a portion of the vein that is a predetermined distance away from the crossing. Therefore, the presence or absence of the arteriovenous crossing phenomenon near the crossing can be detected with higher accuracy.

[0015] Various devices can be used as the fundus image capturing device. As an example, in the present disclosure, a two-dimensional color fundus image captured from the front of the fundus by a fundus camera is used as the fundus image. In this case, a blood vessel image is appropriately acquired based on the color fundus image. However, a two-dimensional fundus image captured from the front of the fundus by a scanning laser ophthalmoscope (SLO) may also be used.

[0016] In the blood vessel image acquisition step, the control unit may acquire the blood vessel image by inputting the fundus image into a mathematical model trained by a machine learning algorithm. In this case, a blood vessel image showing blood vessels with high accuracy is easily acquired. The mathematical model may be trained using a fundus image of the subject's eye captured in the past as input training data, and a blood vessel image showing at least one of an artery and a vein in the fundus image of the input training data as output training data. In this case, the trained mathematical model can appropriately output the blood vessel image based on the input fundus image. In the blood vessel image acquisition step, a blood vessel image of an artery and a blood vessel image of a vein may be acquired separately. Also, one blood vessel image capable of identifying each of an artery and a vein may be acquired.

[0017] However, the method of acquiring the blood vessel image may be changed. For example, the blood vessel image may be generated in response to an instruction input by an operator via an operation unit (such as a mouse).

[0018] The control unit may further execute a thinning step for thinning at least the width of the vein (e.g., to make the width one pixel) on the blood vessel image. In the intersection detection step, the control unit may detect the intersection by processing a thinned image in which at least the width of the vein has been thinned. In this case, it becomes easier to appropriately detect the intersection by simple processing compared to processing a blood vessel image in which the width of the blood vessel has not been thinned.

[0019] In the thinning step, the control unit may thin the widths of both the artery and the vein, which makes it easier and more likely that the intersections will be detected properly.

[0020] The control unit may further execute a blood vessel group identification process. In the blood vessel group identification process, the control unit may detect a branching point at which a blood vessel branches, at least for a vein in the blood vessel image. The control unit may identify each of a plurality of blood vessel parts that branch from the detected branching point and are connected in a single line as a different blood vessel group. In this case, each blood vessel group necessarily has two ends. Therefore, it is appropriately understood whether any two ends are ends of the same blood vessel group. Although details will be described later, when detecting an intersection between a vein and an artery, the identification results of each of the plurality of blood vessel groups are used, making it easier to appropriately detect the intersection. In the present disclosure, a blood vessel part that is connected in a single line without branching (including a single blood vessel part that is originally not branched) is expressed as a blood vessel group. Also, a branching point refers to a point where a blood vessel extending in a single line branches into two or more. Therefore, the blood vessel is identified into three or more blood vessel groups starting from the detected branching point.

[0021] A specific method for detecting a branching point of a blood vessel may be appropriately selected. For example, the control unit may use, as a processing unit for detecting a branching point, nine lattice-shaped pixels arranged in a matrix of three vertically and three horizontally with a pixel of interest, which is one of the blood vessel pixels, at the center in a thinned blood vessel image (in the present disclosure, a blood vessel image in which the width of the blood vessel is one pixel). The control unit may detect the pixel of interest as a branching point when three or more of the eight pixels located around the pixel of interest are pixels of the same type of blood vessel. By performing the above processing with each of the multiple pixels on the thinned blood vessel as the pixel of interest, the branching point of the blood vessel can be easily detected with high accuracy.

[0022] In the blood vessel group identification process, the control unit may execute a process of identifying each of a plurality of blood vessel groups for both arteries and veins. In this case, the blood vessel groups of the arteries and veins are appropriately identified, which makes it easier to further improve the accuracy of the process.

[0023] A blood vessel portion that is connected in a single line without branching and has two ends (including a linear blood vessel portion that is originally not branched) is defined as a blood vessel group. In the crossing detection step, the control unit may detect, among the multiple ends of the blood vessel group of a vein, a pair of ends that satisfy both of the conditions (1) that an artery exists between the two ends and (2) that each of the two ends belongs to a different blood vessel group, as a crossing where a vein and an artery cross. In this case, the crossing is appropriately detected with higher accuracy.

[0024] The control unit may detect, as a crossing, a space between a pair of ends that also satisfies the condition (3) that the distance between the two ends is within a specified distance, in addition to the above-mentioned conditions (1) and (2). At a crossing, the distance between the ends of a pair of veins located on both sides of an artery tends to be close. Therefore, by taking the distance between the two ends into consideration, the detection accuracy of the crossing can be further improved. The processing speed for detecting the crossing can also be easily reduced.

[0025] The control unit may also determine whether or not the end of each of the vein blood vessel groups satisfies the above-mentioned conditions (1) and (2) between the end of interest and the closest end of the vein blood vessel group. In this case, it becomes easier to detect the intersection more accurately.

[0026] The method for detecting the position of the end of a vein blood vessel group (hereinafter, sometimes simply referred to as "vein end") can be appropriately selected. For example, the control unit may use nine lattice-shaped pixels arranged in a matrix of three vertically and three horizontally with a pixel of interest, which is one of the blood vessel pixels, at the center in a thinned blood vessel image (in this disclosure, a blood vessel image in which the width of a blood vessel is one pixel), as a unit of processing for detecting the vein end. When only one pixel out of eight pixels located around the pixel of interest is a vein pixel, the control unit may detect the position of the pixel of interest as the position of the vein end. By performing the above processing with each of the multiple pixels on the thinned blood vessel as the pixel of interest, the position of the vein end can be easily detected with high accuracy.

[0027] A specific method for identifying each of the multiple blood vessel groups can be appropriately selected. For example, the control unit may assign a label to each of the multiple blood vessel groups. In this case, the control unit can appropriately determine whether any two blood vessel groups are the same blood vessel group based on the label assigned to each blood vessel group.

[0028] The control unit may exclude an annular region located at the outer periphery of the image (at least one of the fundus image and the blood vessel image) from the target region for determining whether or not an arteriovenous crossing phenomenon occurs in the crossing phenomenon determination step. The brightness of the outer periphery of the fundus image is often darker than the brightness of the central portion. Furthermore, blood vessels in the outer periphery of the fundus tend to be thinner than blood vessels in the central portion. Therefore, if the outer periphery of the image is included in the detection target, the detection accuracy of the presence or absence of an arteriovenous crossing phenomenon is likely to decrease. In contrast, by excluding the outer periphery of the image from the detection target, the presence or absence of an arteriovenous crossing phenomenon can be detected with higher accuracy.

[0029] The control unit may exclude a part of the fundus image including the optic disc (hereinafter, sometimes simply referred to as "optic disc") from the target area for determining whether or not the arteriovenous crossing phenomenon occurs in the crossing phenomenon determination step. In the vicinity of the optical disc in the fundus, the arteries and veins are more complicatedly intertwined than in other areas, so it is often difficult to appropriately detect the presence or absence of the arteriovenous crossing phenomenon. Therefore, by excluding the part of the area including the optical disc from the detection target, the presence or absence of the arteriovenous crossing phenomenon can be more easily detected with higher accuracy.

[0030] The control unit may input the fundus image into a mathematical model trained by a machine learning algorithm to specify the position of the papilla in the blood vessel image. The control unit may exclude a part of the region including the specified position of the papilla (e.g., a circular region of a predetermined size centered on the position of the papilla) from the region to be determined for the presence or absence of the arteriovenous crossing phenomenon. In this case, the position of the papilla is easily specified with high accuracy. In addition, when the machine learning algorithm is used for acquiring the blood vessel image, both the blood vessel image and the position of the papilla may be output by the same mathematical model, or may be output separately by different mathematical models. However, it is also possible to change the method of specifying the position of the papilla. For example, the control unit may specify the position of the papilla by performing known image processing on the fundus image or the blood vessel image.

[0031] The control unit may exclude veins in the fundus image whose thickness on the blood vessel image is equal to or less than a threshold from target veins for determining whether or not an arteriovenous crossing phenomenon occurs in the crossing phenomenon determination step. If the thickness of a vein is too thin, the detection accuracy of the arteriovenous crossing phenomenon is likely to decrease. Therefore, by excluding veins whose thickness is equal to or less than a threshold from the target, the detection accuracy of the arteriovenous crossing phenomenon is likely to be appropriately improved.

[0032] The control unit may also display the detected position of the crossing in an identifiable manner on the fundus image. In this case, a doctor or the like can easily grasp the position of the detected crossing, and therefore can appropriately grasp the state of the vein at the crossing (for example, whether or not the arteriovenous crossing phenomenon actually occurs).

[0033] In the crossing phenomenon determination step, the control unit may determine whether or not an arteriovenous crossing phenomenon occurs for each of the pair of attention regions, which are the crossing portion of the fundus image and extend from a pair of contours of an artery in different directions along a vein, as an attention region. An arteriovenous crossing phenomenon may occur only in one of a pair of veins extending in different directions from an artery. Therefore, by performing a process of detecting the presence or absence of an arteriovenous crossing phenomenon separately for the pair of attention regions, the arteriovenous crossing phenomenon can be appropriately detected even if the arteriovenous crossing phenomenon occurs only in one of the attention regions.

[0034] The control unit may further execute an external pixel information acquisition step of acquiring external pixel information, which is pixel information outside a position in the blood vessel image that is determined to be a vein, from the fundus image. In the crossover phenomenon determination step, the control unit may determine whether or not an arteriovenous crossover phenomenon occurs at the crossover portion by comparing pixel information within a region of interest in the fundus image with the vein pixel information and the external pixel information.

[0035] As described above, if no arteriovenous intersection occurs, the pixel information in the attention area of ​​the fundus image is likely to approximate the vein pixel information of the area away from the intersection. On the other hand, if the arteriovenous intersection occurs and the color of the vein near the intersection is light, the pixel information in the attention area of ​​the fundus image is likely to approximate the pixel information outside the position that is determined to be a vein by the blood vessel image. Therefore, by comparing the external pixel information in addition to the vein pixel information with the pixel information in the attention area, the presence or absence of the arteriovenous intersection can be detected with higher accuracy. For example, even if the color of each fundus image differs due to the individual difference in the color of the fundus and the type of device used to capture the fundus image, the detection accuracy of the arteriovenous intersection can be appropriately improved by using both the vein pixel information and the external pixel information.

[0036] In the crossover phenomenon determination step, the control unit may determine whether pixel information in the attention area in the fundus image is closer to vein pixel information or external pixel information. The control unit may determine whether arteriovenous crossover phenomenon occurs depending on whether the degree to which the pixel information is determined to be closer to the external pixel information is equal to or greater than a threshold. In this case, whether arteriovenous crossover phenomenon occurs is determined depending on whether there are more pixels closer to the external pixel information among the multiple pixels in the attention area. This makes it easier to further improve the detection accuracy of the arteriovenous crossover phenomenon.

[0037] In the external pixel information acquisition step, the control unit may detect an end of a portion of a vein that appears in the fundus image as if it is divided by crossing with an artery at a crossing point. The control unit may acquire pixel information of a portion adjacent to the outside of the end of the vein at the crossing point as the external pixel information. The effect of the color change of a vein due to the arteriovenous crossing phenomenon is likely to appear around the end of the vein at the crossing point (which is not actually an end, but looks like an end because it crosses with an artery) at the crossing point. Therefore, by acquiring pixel information of a portion adjacent to the outside of the end of the vein at the crossing point as the external pixel information, it becomes easier to further improve the detection accuracy of the arteriovenous crossing phenomenon.

[0038] However, the method of acquiring the external pixel information may be changed. For example, the control unit may acquire pixel information of a portion of the fundus image other than the artery and the vein as the external pixel information. The control unit may also acquire an average value of pixel information of a bright area in the fundus image as the external pixel information.

[0039] The control unit can also determine the presence or absence of an arteriovenous crossing by comparing only the vein pixel information with the pixel information in the region of interest without using external pixel information. For example, the control unit may determine whether or not the difference between the pixel information in the region of interest and the vein pixel information is equal to or greater than a reference value. The control unit may determine whether or not an arteriovenous crossing has occurred based on the degree to which the difference between the vein pixel information is determined to be equal to or greater than the reference value. Even in this case, the arteriovenous crossing is appropriately detected.

[0040] Each pixel of the fundus image may be expressed by the three primary colors of RGB. In the crossing phenomenon determination step, the control unit may determine whether or not the arteriovenous crossing phenomenon occurs based on pixel information excluding the B value among the R value, G value, and B value. As a result of repeated trials and studies by the inventor of the present invention, it has been newly discovered that the B value in the fundus image has little effect on the difference between the color of the vein and the color of the part other than the blood vessel. By detecting the presence or absence of the arteriovenous crossing phenomenon based on pixel information excluding the B value, it becomes easier to further improve the detection accuracy. Furthermore, the effect of improving the speed of the detection process may also be obtained.

[0041] However, all of the R value, G value, and B value may be used to determine the presence or absence of the arteriovenous intersection phenomenon. Also, pixel information other than RGB values ​​(e.g., at least one of brightness, saturation, etc.) may be used. The color system expressing the fundus image may be a color system other than the RGB color system (e.g., the CMYK color system, the LAB color system, etc.).

[0042] In the crossing phenomenon determination step, the control unit may move a search line, which has a shape that follows the contour of the artery at the crossing and is set to a length equal to or less than the width of the vein at the crossing, from the contour of the artery in a direction along the vein, to include the area through which the search line passes in the attention area. In this case, the shape of the attention area becomes an appropriate shape that spreads from the contour of the artery along the vein. This makes it easier to further improve the detection accuracy of the arteriovenous crossing phenomenon.

[0043] In the crossing phenomenon determination step, the control unit may set a straight line connecting the ends of a pair of veins extending in different directions from the crossing in the fundus image on the crossing side. The control unit may include in the attention area a passing area of ​​the search line when the search line is moved along a reference straight line from the contour line of the artery to an end (in the present disclosure, the end closer to the contour line of the pair of ends located on both sides of the crossing). In the vicinity of the crossing, an actually existing vein may not appear in the blood vessel image due to the influence of the arteriovenous crossing phenomenon, etc. The control unit can appropriately set the attention area even in the vicinity of the crossing where the vein is difficult to appear in the blood vessel image by setting a reference straight line connecting the ends of the pair of veins and moving the search line along the reference straight line.

[0044] The control unit may further execute a thinning step of thinning at least the width of the vein on the blood vessel image. In the crossing phenomenon determination step, the control unit may detect each end of the pair of veins based on the positions of the end points of the pair of veins extending in different directions from the crossing, among the veins thinned in the thinning step. In this case, each end of the pair of veins near the crossing can be easily detected with high accuracy based on the thinned image.

[0045] In the crossing phenomenon determination step, the control unit may determine the position of the vein end point to be a position that is a predetermined distance (e.g., 10 pixels) along the thinned vein from the end point of the vein thinned in the thinning step. The accuracy of the vein thinning process is likely to decrease near the end point of the vein compared to a position away from the end point. By determining the position of the end point to be a predetermined distance along the vein from the end point of the thinned vein rather than the end point of the thinned vein, the control unit can set a more appropriate position for the position of the vein end point.

[0046] <Embodiment> (Device configuration) A typical embodiment of the present disclosure will be described below with reference to the drawings. As shown in FIG. 1, in this embodiment, a fundus image processing device 1 and a fundus image photographing device 11 are used. The fundus image photographing device 11 photographs a fundus image of a subject's eye (in this embodiment, a two-dimensional front image of the fundus viewed from a direction along the photographing light). The fundus image processing device 1 processes the fundus image photographed by the fundus image photographing device 11 to automatically detect the presence or absence of an arteriovenous crossing phenomenon in the fundus depicted in the fundus image.

[0047] As an example, a personal computer (hereinafter, referred to as "PC") is used as the fundus image processing device 1 of this embodiment. However, the device capable of functioning as the fundus image processing device 1 is not limited to a PC. For example, the fundus image photographing device 11 or a server or the like may function as the fundus image processing device 1. When the fundus image photographing device 11 functions as the fundus image processing device 1, the fundus image photographing device 11 can automatically detect the presence or absence of arteriovenous crossing phenomenon in the fundus shown in the fundus image based on the photographed fundus image while photographing the fundus image. In addition, control units of multiple devices (for example, the CPU of the PC and the CPU of the fundus image photographing device) may cooperate to execute the fundus image processing described later.

[0048] In addition, in this embodiment, a CPU is used as an example of a controller that performs various processes. However, it goes without saying that a controller other than a CPU may be used for at least a part of the various devices. For example, a GPU may be used as a controller to speed up the processing.

[0049] The fundus image processing device 1 will be described. The fundus image processing device 1 includes a control unit 2 that performs various control processes, and a communication I / F 5. The control unit 2 includes a CPU 3 that is a controller responsible for control, and a storage device 4 that can store programs, data, and the like. The storage device 4 may store a fundus image processing program for executing fundus image processing (see FIG. 3) described below. The communication I / F 5 also connects the fundus image processing device 1 to other devices (e.g., a fundus image photographing device 11, etc.).

[0050] The fundus image processing device 1 is connected to an operation unit 7 and a display device 8. The operation unit 7 is operated by a user to input various instructions to the fundus image processing device 1. For example, at least one of a keyboard, a mouse, a touch panel, etc. can be used for the operation unit 7. Note that a microphone or the like for inputting various instructions may be used together with the operation unit 7 or instead of the operation unit 7. The display device 8 displays various images. For the display device 8, various devices capable of displaying images (for example, at least one of a monitor, a display, a projector, etc.) can be used.

[0051] The fundus image processing device 1 can acquire fundus image data (hereinafter, sometimes simply referred to as "fundus image") from the fundus image photographing device 11. The fundus image processing device 1 may acquire fundus image data from the fundus image photographing device 11 by at least one of wired communication, wireless communication, a removable storage medium (e.g., USB memory), and the like.

[0052] The fundus image photographing device 11 will be described. Various devices for photographing a fundus image of a subject's eye can be used as the fundus image photographing device 11. As an example, the fundus image photographing device 11 used in this embodiment is a fundus camera capable of photographing a two-dimensional color front image of the fundus using visible light. Therefore, various processes described later are appropriately performed based on the color fundus image. However, devices other than a fundus camera (for example, a laser scanning ophthalmoscope (SLO) capable of photographing a color fundus image, etc.) may be used as the fundus image photographing device.

[0053] The fundus image capturing device 11 includes a control unit 12 that performs various control processes, and a fundus image capturing section 16. The control unit 12 includes a CPU 13 that is a controller responsible for control, and a storage device 14 that can store programs, data, and the like. The fundus image capturing section 16 includes optical members and the like for capturing a fundus image of the subject's eye. When the fundus image capturing device 11 performs at least a part of the fundus image processing (see FIG. 3) described below, it goes without saying that at least a part of a fundus image processing program for performing the fundus image processing is stored in the storage device 14.

[0054] With reference to FIG. 2, an example of a method in which the fundus image processing device 1 of the present embodiment acquires a blood vessel image will be described. A blood vessel image is an image showing blood vessels (arteries and veins) included in a fundus image. The fundus image processing device 1 of the present embodiment acquires a blood vessel image showing blood vessels in the input fundus image by inputting the fundus image into a mathematical model trained by a machine learning algorithm. The fundus image processing device 1 of the present embodiment also inputs a fundus image into a mathematical model trained by a machine learning algorithm to specify the position (more specifically, the position of the center of gravity of the optic disc) of the input fundus image by inputting the fundus image into the mathematical model trained by a machine learning algorithm. The mathematical model is trained in advance to output a blood vessel image and the position of the disc for the input fundus image when a fundus image is input. The mathematical model that outputs a blood vessel image and the mathematical model that outputs the position of the disc may be constructed separately.

[0055] The mathematical model is constructed to output blood vessel images and the location of the nipple by being trained with a training data set, which includes input data (input training data) and output data (output training data).

[0056] FIG. 2 shows an example of a fundus image 30 and a blood vessel image 40 (40A, 40B). In this embodiment, the fundus image 30, which is a two-dimensional color front image captured by the fundus image capturing device 11, is used as the input training data. In this embodiment, the image area of ​​the fundus image 30 used as the input training data includes both the optic disc 31 and the macula 32 of the subject's eye. In addition, blood vessel images 40A and 40B, which are images showing at least one of an artery and a vein in the fundus image 30 used as the input training data, and information showing the position of the optic disc (in this embodiment, the position G of the center of gravity of the optic disc) are used as the output training data. In this embodiment, the blood vessel image 40A of the artery in the fundus image 30 and the blood vessel image 40B of the vein in the fundus image 30 are included in the output training data. Therefore, when the fundus image 30 is input, the constructed mathematical model can output a blood vessel image 40 showing the blood vessels included in the input fundus image 30. 2 illustrates a case where an arterial blood vessel image 40A and a venous blood vessel image 40B are acquired separately. However, a single blood vessel image including both an artery and a vein may be acquired in a state where the artery and the vein are distinguishable from each other. The output training data (i.e., the blood vessel images 40A and 40B and information indicating the position of the papilla) may be generated, for example, according to an instruction input by an operator who has checked the fundus image 30, which is the input training data.

[0057] (Fundus image processing) The fundus image processing performed by the fundus image processing device 1 of this embodiment will be described with reference to Fig. 3 to Fig. 13. As described above, in the fundus image processing, the fundus image photographed by the fundus image photographing device 11 is processed to automatically detect the presence or absence of arteriovenous intersection in the fundus shown in the fundus image. The fundus image processing illustrated in Fig. 3 is executed by the CPU 3 of the fundus image processing device 1 in accordance with a fundus image processing program stored in the storage device 4.

[0058] 3, the CPU 3 acquires (S11) a fundus image 30 (see FIG. 2), which is a two-dimensional color front image captured by a fundus image capturing device (fundus camera in this embodiment) 11. The fundus image 30 includes blood vessels (arteries and veins) at the fundus of the subject's eye.

[0059] Next, the CPU 3 acquires a blood vessel image 40 showing the arteries and veins included in the fundus image 30 acquired in S1 (S2). In this embodiment, the CPU 3 inputs the fundus image 30 into a mathematical model trained by a machine learning algorithm, thereby acquiring the blood vessel image 40 output by the mathematical model. Therefore, it is easy to acquire a blood vessel image 40 showing the blood vessels in the fundus image 30 with high accuracy. Note that, in FIG. 2, for ease of understanding, a case is illustrated in which a blood vessel image 40A of an artery and a blood vessel image 40B of a vein included in the fundus image 30 are acquired separately. However, in this embodiment, a single blood vessel image 40 is acquired in which the arteries and veins are each shown in a distinguishable state.

[0060] The CPU 3 identifies the position of the papilla in the blood vessel image 40 acquired in S2 (in this embodiment, the position of the center of gravity of the papilla) (S3). In this embodiment, the CPU 3 inputs the fundus image 30 to a mathematical model trained by a machine learning algorithm, and acquires the position of the papilla in the fundus image 30 output by the mathematical model. Therefore, the position of the papilla in the fundus image 30 and the blood vessel image 40 can be easily identified with high accuracy.

[0061] The CPU 3 executes thinning processing for thinning at least the width of the vein (in this embodiment, the width of each of the vein and the artery) on the blood vessel image 40 acquired in S2 (S4). As shown in FIG. 4, in S4 of this embodiment, the CPU 3 generates a thinned image 41 by setting the width of each of the artery 51A and the vein 51B included in the blood vessel image 40 to one pixel. The generated thinned image 41 shows a thinned artery 52A and a thinned vein 52B. The thinned artery 52A coincides with the center position in the width direction of the artery 51A in the blood vessel image 40. The thinned vein 52B coincides with the center position in the width direction of the vein 51B in the blood vessel image 40. By using the thinned image 41 in various processes described later, the processes are simplified and the processing accuracy is easily improved.

[0062] The CPU 3 executes a blood vessel group identification process (S5). A blood vessel group is a blood vessel portion connected in a single line without branching (including a single blood vessel that is originally not branched). In the blood vessel group identification process, at least each portion of the veins (in this embodiment, each of the veins and the arteries) in the blood vessel image 40 is identified as a plurality of blood vessel groups. As will be described in detail later, when detecting an intersection 70 between a vein and an artery (see FIG. 7), the identification results of each of the plurality of blood vessel groups are used, making it easier to appropriately detect the intersection 70.

[0063] An example of a processing method of the blood vessel group identification process will be described below. When the CPU 3 starts the process of S5, it detects a branching point where blood vessels (in this embodiment, each of an artery and a vein) branch off. As shown in FIG. 5, the CPU 3 of this embodiment sets one of the blood vessel pixels in the thinned image 41 (that is, a blood vessel image in which the width of the blood vessel is set to one pixel) acquired in S4 as a target pixel 55. The CPU 3 sets nine lattice-like pixels arranged in three rows and three columns with the target pixel 55 at the center as a processing unit for detecting a branching point. The CPU 3 calculates the number of pixels of the same type of blood vessel (artery or vein) in eight pixels located around the target pixel 55. If the target pixel 55 is in the center of a linear blood vessel, two pixels of the same type of blood vessel will be included in the eight pixels. If the target pixel 55 is at the end of a blood vessel, only one pixel of the same type of blood vessel will be included in the eight pixels. If the target pixel 55 is a branching point, three or more pixels of the same type of blood vessel will be included in the eight pixels. Therefore, as shown in FIG. 5, when three or more pixels of the same type of blood vessel are included among eight pixels positioned around the pixel of interest 55, the CPU 3 detects the pixel of interest 55 as a branch point.

[0064] When the CPU 5 detects the pixel of interest 55 as a branch point, the CPU 5 executes a branch point deletion process to delete the pixel of the blood vessel that is the pixel of interest 55. As a result, as shown in FIG. 5, a blood vessel extending in three or more different directions starting from a branch point is divided into a plurality of blood vessel groups. The CPU 5 executes a labeling process to assign a label to each of the divided blood vessel groups. As a result, each blood vessel group is appropriately identified by the assigned label. In the example shown in FIG. 6, one blood vessel that branched from a branch point is divided into three blood vessel groups 53S, 53T, and 53U by the branch point deletion process. Furthermore, the label "1" is assigned to the blood vessel group 53S, the label "2" is assigned to the blood vessel group 53T, and the label "3" is assigned to the blood vessel group 53U. In addition, in S5 of this embodiment, a label is also assigned to a single linear blood vessel that was not originally branched.

[0065] Returning to the explanation of FIG. 3, the CPU 3 processes the blood vessel image 40 to detect at least one of the crossings 70 where an artery and a vein cross in the fundus image 30 (S6). As a result, an arteriovenous crossing phenomenon detection process (S9) described later is appropriately performed on the crossings 70 detected in S6. Note that in S6 of this embodiment, the CPU 3 performs the crossing detection process based on a thinned image 41 (see FIGS. 4 and 7) generated by processing the blood vessel image 40. However, the crossings 70 may be detected directly based on the blood vessel image 40.

[0066] In the arteriovenous crossing detection process (S9) described later, pixel information in the attention area AR near the crossing 70 is compared with pixel information in other areas to detect the presence or absence of the arteriovenous crossing. However, the brightness of the outer periphery of the fundus image 30 is often darker than that of the center. In the dark areas of the fundus image 30, the difference between the pixel information in the attention area AR and the pixel information in other areas is unlikely to occur, so the detection accuracy of the presence or absence of the arteriovenous crossing may decrease. Furthermore, blood vessels in the outer periphery of the fundus tend to be thinner than blood vessels in the center. In areas where the blood vessels are thin, the detection accuracy of the presence or absence of the arteriovenous crossing tends to decrease. Therefore, in S6 of this embodiment, the CPU 3 excludes an annular area located in the outer periphery of the blood vessel image 40 (thinned image 41 generated from the blood vessel image 40) from the target area for detecting the crossing 70. As a result, in the arteriovenous crossing detection process (S9) described later, the annular area located on the outer periphery of the fundus image 30 is excluded from the area to be detected for the presence or absence of the arteriovenous crossing, which makes it easier to detect the presence or absence of the arteriovenous crossing with higher accuracy.

[0067] In addition, since arteries and veins are more complicatedly intertwined in the vicinity of the optic nerve head in the fundus than in other regions, it is often difficult to appropriately detect the presence or absence of an arteriovenous crossing phenomenon. Therefore, in S6 of this embodiment, the CPU 3 excludes a part of the blood vessel image 40 (thinned image 41 generated from the blood vessel image 40) including the part where the optic nerve head is located from the target region for detecting the crossing portion 70. As an example, the CPU 3 excludes a circular region having a predetermined radius centered on the position of the optic nerve head detected in S3 (the position of the center of gravity of the optic nerve head 3 in this embodiment) from the target region for detecting the crossing portion 70. As a result, in the arteriovenous crossing phenomenon detection process (S9) described later, a part of the fundus image 30 including the optic nerve head is excluded from the target region for detecting the presence or absence of an arteriovenous crossing phenomenon. Therefore, the presence or absence of an arteriovenous crossing phenomenon can be detected more accurately.

[0068] Furthermore, in S6 of this embodiment, the CPU 3 detects the thickness of each of the multiple veins (in this embodiment, a blood vessel group of multiple veins) shown in the blood vessel image 40 by image processing or the like. The CPU 3 excludes veins (a blood vessel group of multiple veins) shown in the blood vessel image 40 that have a thickness equal to or less than a threshold from the target veins for detecting the crossing portion 70. As a result, in the arteriovenous crossing detection process (S9) described later, veins shown in the fundus image 30 that have a thickness equal to or less than a threshold are also excluded from the target veins for detecting the presence or absence of an arteriovenous crossing. This makes it easier to detect the presence or absence of an arteriovenous crossing with higher accuracy.

[0069] The details of the intersection detection process (S6) in this embodiment will be described with reference to Fig. 7. As an example, in S6 of this embodiment, the thinned image 41 generated from the blood vessel image 40 in S4 and the identification results of each of the multiple blood vessel groups 53 acquired in S5 are used. As described above, a blood vessel group 53 is a blood vessel portion that is connected in a single line without branching (including one blood vessel that is originally not branched). Each blood vessel group 53 includes two ends 60.

[0070] An example of a process for detecting the end 60 of a vein will be described. In this embodiment, the CPU 3 sets one of the pixels of the vein in the thinned image 41 acquired in S4 (i.e., a blood vessel image in which the width of the blood vessel is one pixel) as a pixel of interest. The CPU 3 sets nine lattice-shaped pixels arranged in three rows and three columns with the pixel of interest at the center as a unit of the process for detecting the end 60 of the vein. The CPU 3 calculates the number of vein pixels in eight pixels located around the pixel of interest. As described above, if the pixel of interest is the end of a vein, only one vein pixel will be included in the eight pixels. Therefore, the CPU 3 detects the pixel of interest as the end of a vein when only one vein pixel is included in the eight pixels located around the pixel of interest.

[0071] The CPU 3 detects a pair (two) of vein ends 60 that satisfy both of the following conditions (1) and (2) by searching for ends 60 of vein blood vessel groups 53 in the thinned image 41: (1) An artery (thinned artery 52A in this embodiment) exists between the two ends 60. (2) The two ends 60 each belong to a different blood vessel group 53. The CPU 3 can appropriately detect the intersection 70 with high accuracy by detecting the intersection 70 between the ends 60 of a pair of veins that satisfy both conditions (1) and (2).

[0072] Furthermore, at the crossing 70, the distance between the ends 60 of a pair of veins located on either side of the artery becomes close. Therefore, in this embodiment, the CPU 3 detects, as the crossing 70, the distance between the pair of ends 60 that satisfies the following condition (3) in addition to the above-mentioned conditions (1) and (2): (3) The distance between the two ends 60 is within a specified distance.

[0073] In the example shown in FIG. 7, (1) a thinned artery 52A exists between end 60X and end 60Y on thinned image 41. (2) The blood vessel group 53X to which end 60X belongs is different from the blood vessel group 53Y to which end 60Y belongs. Furthermore, (3) the distance between end 60X and end 60Y is within a specified distance. Therefore, CPU 3 detects the area between end 60X and end 60Y as an intersection 70. Note that, when there are multiple intersections 70 that satisfy the condition, CPU 3 may detect each of the multiple intersections 70.

[0074] It is also possible to change the method of detecting the intersection 70. For example, the CPU 3 may determine whether or not both of the above-mentioned conditions (1) and (2) are satisfied between the end 60 of interest (e.g., end 60X in FIG. 7) and another end 60 (60Y in FIG. 7) closest to the end 60 of interest. In this case as well, the intersection 70 is appropriately detected between a pair of ends 60 that are close to each other.

[0075] In this embodiment, the CPU 3 displays the position detected as the intersection 70 in an identifiable manner on the fundus image 30. Therefore, a doctor or the like can easily grasp the position of the detected intersection 70 on the fundus image 30, and can appropriately grasp the state of the vein at the intersection 70 (for example, whether or not an arteriovenous intersection phenomenon actually occurs) by himself or herself.

[0076] Returning to the description of FIG. 3, the CPU 3 acquires pixel information (hereinafter referred to as "vein pixel information") of a portion of a vein that is separated from the intersection 70 detected in S6 among the veins captured in the fundus image 30 (see FIG. 2 and FIG. 9) (S7). FIG. 8 shows an example of a vein pixel information acquisition position 81 on the blood vessel image 40. In S7 of the present embodiment, the CPU 3 sets the vein pixel information acquisition position 81 at a portion of a vein that is separated from the intersection 70 by a predetermined distance (e.g., 50 pixels, etc.). As shown in FIG. 9, the CPU 3 acquires, as vein pixel information, pixel information of a position corresponding to the vein pixel information acquisition position 81 set on the blood vessel image 40 among the fundus image 30 (color fundus image) acquired in S1. In the arteriovenous crossing detection process described below (see FIG. 9), pixel information within an area of ​​interest AL in the vein near the crossing 70 (pixel information of an area that is susceptible to the influence of arteriovenous crossing) is compared with pixel information of a portion of the vein away from the crossing 70 (pixel information of an area that is not easily affected by arteriovenous crossing), making it easier to detect the presence or absence of an arteriovenous crossing with high accuracy.

[0077] Note that, if vein pixel information is acquired from a wide area of ​​a vein portion away from the intersection 70, the burden of acquiring the pixel information increases. Therefore, in this embodiment, a linear vein pixel information acquisition position 81 is set in a vein portion away from the intersection 70. By acquiring pixel information of pixels at the linear vein pixel information acquisition position 81, the processing burden of the pixel information is appropriately reduced.

[0078] The CPU 3 acquires pixel information (hereinafter referred to as "external pixel information") from the fundus image 30 (see Figs. 2 and 9) outside the position determined to be a vein in the blood vessel image 40 (S8). If no arteriovenous crossing occurs, the pixel information of the vein near the crossing 70 in the fundus image 30 is likely to approximate the vein pixel information of a portion distant from the crossing 70. On the other hand, if an arteriovenous crossing occurs and the color of the vein near the crossing 70 is light, the pixel information of the vein near the crossing 70 is likely to approximate the pixel information outside the position determined to be a vein in the blood vessel image 40. Therefore, in the arteriovenous crossing detection process (see Fig. 9) described later, the external pixel information in addition to the vein pixel information is compared with the pixel information in the region of interest AL, making it easier to detect the presence or absence of an arteriovenous crossing with higher accuracy.

[0079] FIG. 10 shows an example of the external pixel information acquisition position 82 on the blood vessel image 40. In S8 of the present embodiment, the CPU 3 processes the blood vessel image 40 and the thinned image 41 to detect the end of the vein on the crossing part 70 side of the portion of the vein shown in the fundus image 30 that appears in the fundus image 30 and the blood vessel image 40 as if it is divided by crossing the artery at the crossing part 70. The CPU 3 sets the external pixel information acquisition position 82 in the blood vessel image 40 at a portion adjacent to the outside of the end of the vein at the crossing part 70. As shown in FIG. 9, the CPU 3 acquires pixel information of the position corresponding to the external pixel information acquisition position 82 set on the blood vessel image 40 in the fundus image 30 (color fundus image) acquired in S1 as external pixel information. The effect of the change in color of the vein due to the arteriovenous crossing phenomenon is likely to appear around the end of the vein at the crossing part 70 (which is not actually an end, but looks like an end by crossing the artery). Therefore, by acquiring pixel information of a region adjacent to the outside of the end of the vein at the crossing 70 as external pixel information, it becomes easier to further improve the detection accuracy of the arteriovenous crossing phenomenon.

[0080] However, the method of acquiring the external pixel information may be changed. For example, the CPU 3 may acquire pixel information of a portion of the fundus image 30 other than the artery and the vein as the external pixel information. The CPU 3 may also acquire an average value of pixel information of a bright region in the fundus image 30 as the external pixel information.

[0081] Next, the CPU 3 executes an arteriovenous crossing detection process (S9). In the process of S9, an area of ​​the crossing 70 of the fundus image 30 that spreads from the contour of the artery along the vein is set as an attention area AL. The pixel information in the attention area AL in the fundus image 30 is compared with the vein pixel information (in this embodiment, both the vein pixel information and the external pixel information) to detect the presence or absence of the arteriovenous crossing at the crossing 70.

[0082] When arteriosclerosis occurs, the vein at the crossing 70 may be crushed by the artery in the depth direction of the fundus, and the thickness may become smaller. In this case, the amount of blood in the depth direction of the vein at the crossing 70 is reduced compared to the vein that is not crushed in the depth direction. Therefore, when the vein at the crossing 70 is crushed in the depth direction, it is likely to appear lighter in color than the vein that is not crushed in the depth direction, regardless of whether it appears to be constricted or not. In other words, when the color of the vein near the crossing 70 is lighter due to the effect of the arteriovenous crossing phenomenon, and when the vein is constricted, etc., it is difficult to approximate the pixel information in the attention area AL in the fundus image 30 to the vein pixel information. On the other hand, if the arteriovenous crossing phenomenon does not occur, the vein crosses the artery at the crossing 70 with a substantially constant thickness. As a result, the pixel information in the attention area AL in the fundus image 30 is likely to approximate the vein pixel information of a part away from the crossing 70. Therefore, according to the fundus image processing device 1 of this embodiment, it becomes easier to properly detect the occurrence of arteriovenous crossing phenomenon not only when the vein is narrowed, but also when the color of the vein near the crossing point 70 is lighter, etc.

[0083] As described above, in S9 of the present embodiment, the presence or absence of an arteriovenous crossing phenomenon can be detected with higher accuracy by comparing the external pixel information with the pixel information in the attention area AL in addition to the vein pixel information. For example, even if there is a difference in color between the fundus images 30 due to individual differences in fundus color and the type of device used to capture the fundus images 30, the detection accuracy of the arteriovenous crossing phenomenon can be appropriately improved by using both the vein pixel information and the external pixel information.

[0084] In detail, in S9 of the present embodiment, the CPU 3 judges whether the pixel information in the attention area AL in the fundus image 30 is closer to the vein pixel information or the external pixel information. The CPU 3 judges whether the arteriovenous crossing phenomenon occurs depending on whether the degree to which the pixel information is judged to be closer to the external pixel information is equal to or greater than a threshold value. In this case, whether the arteriovenous crossing phenomenon occurs is judged depending on whether there are many pixels closer to the external pixel information among the multiple pixels in the attention area AL. This makes it easier to further improve the detection accuracy of the arteriovenous crossing phenomenon.

[0085] In addition, each pixel of the fundus image 30 used in this embodiment is expressed by the three primary colors of RGB. In S9 of this embodiment, the CPU 3 judges whether or not the arteriovenous crossing phenomenon occurs based on pixel information excluding the B value among the R value, G value, and B value. As a result of repeated trials and studies by the inventor of the present invention, it has been newly discovered that the B value in the fundus image 30 has little effect on the difference between the color of the vein and the color of the part other than the blood vessel. By detecting the presence or absence of the arteriovenous crossing phenomenon based on pixel information excluding the B value, it becomes easier to further improve the detection accuracy. Furthermore, the effect of improving the speed of the detection process may also be obtained.

[0086] An example of a processing method of the arteriovenous crossing detection process (S9) will be described with reference to Figs. 11 to 13. As shown in Fig. 11, in this embodiment, the CPU 3 sets a search line SL on the contour of the artery in the fundus image 30, the search line SL being shaped along the contour of the artery (artery 51A in the blood vessel image 40 in Fig. 11) at the crossing portion 70 and set to a length equal to or less than the width of the vein (vein 51B in the blood vessel image 40 in Fig. 11) at the crossing portion 70. The length of the search line SL may be the length in a direction intersecting with the moving direction of the search line SL, which will be described later. As described above, in this embodiment, among the veins appearing in the fundus image 30, veins having a thickness equal to or less than a threshold value are excluded from the target veins for detecting the presence or absence of the arteriovenous crossing phenomenon. Therefore, the length of the search line SL in the direction intersecting with the moving direction may be set to a threshold value or less for excluding thin veins from the target veins.

[0087] As shown in Fig. 11 to Fig. 13, the CPU 3 moves the set search line SL in the direction along the vein from the contour line of the artery within the fundus image 30. The CPU 3 includes the area through which the search line SL passes in the fundus image 30 in the attention area AL (AL1 and AL2). As a result, the shape of the attention area AL becomes an appropriate area that spreads from the contour line of the artery along the vein.

[0088] As an example, in this embodiment, the CPU 3 acquires pixel information (e.g., average value, etc.) of a plurality of pixels located on the search line SL among a plurality of pixels in the fundus image 30, each time the search line SL is moved by one pixel. The CPU 3 determines whether the pixel information of the pixel on the search line SL is closer to the vein pixel information or the external pixel information by using the sum of squares of the differences in the pixel information, etc. If the pixel information of the pixel on the search line SL is closer to the external pixel information than the vein pixel information, the memory device stores that the pixel information of the portion where the search line SL is located at that time is closer to the external pixel information. The above process is executed each time the search line SL is moved by one pixel. When the movement of the search line SL is completed, the CPU 3 determines that an arteriovenous crossing phenomenon occurs at the crossing portion 70 if the number of times that the pixel information of the portion where the search line SL is located has been close to the external pixel information is equal to or greater than a predetermined number of times.

[0089] In addition, in the vicinity of the intersection 70, an artery and a vein cross each other, so that an actually existing vein may not appear in the fundus image 30 and the blood vessel image 40. In this case, it is difficult for the CPU 3 to move the search line SL along the vein in the vicinity of the intersection 70. Therefore, as shown in FIG. 11, the CPU 3 sets ends 521P and 522P on the intersection 70 side of each of a pair of veins (a pair of veins 51B in the blood vessel image 40 in FIG. 11) extending in different directions from the intersection 70 in the fundus image 30 (in this embodiment, the blood vessel image 40 or the thinned image 41 generated based on the fundus image 30). The CPU 3 sets a reference straight line BL connecting between the set pair (two) ends 521P and 522P. As shown in FIG. 12, the CPU 3 moves the search line SL from the contour line of the artery to one end 521P along the set reference straight line BL. The CPU 3 includes in the attention area AL1 the search line SL passes through when the search line SL is moved from the contour line of the artery to one end 521P. As a result, the attention area AL is appropriately set even in the vicinity of the intersection 70 where veins are difficult to appear in the image.

[0090] 12 and 13, after the search line SL reaches one end 521P, the CPU 3 further moves the search line SL along the vein (thinned vein 52B in this embodiment) a predetermined distance. The CPU 3 includes in the attention area AL a passing area AR2 (see FIG. 13) of the search line SL when the search line SL is moved along the thinned vein 52B.

[0091] As shown in Fig. 11, the CPU 3 sets the search line SL such that the center of the search line SL in a direction intersecting the movement direction (hereinafter simply referred to as "the center of the search line SL") is located on the reference line BL. As shown in Fig. 12, the CPU 3 moves the search line SL along the reference line BL while the center of the search line SL is aligned with the reference line BL. Thereafter, as shown in Fig. 13, the CPU 3 moves the search line SL along the reference line BL while the center of the search line SL is aligned with the thinned vein 52B. As a result, the attention area AL can be more appropriately set.

[0092] In this embodiment, the CPU 3 sets the positions of the vein ends 521P, 522P based on the positions of each of the endpoints of a pair of thinned veins 52B extending in different directions from the intersection 70 among the thinned veins 52B included in the thinned image 41 generated in S4. As a result, the ends of each of the pair of veins at the intersection 70 can be set with higher accuracy.

[0093] 11 to 13, the accuracy of thinning is likely to decrease near the end points of the thinned vein 52B itself generated by the thinning process, compared to positions away from the end points. Therefore, in this embodiment, the CPU 3 sets positions that are a predetermined distance (e.g., 10 pixels) away from the end points of the thinned vein 52B itself along the thinned vein 52B as the positions of the vein ends 521P, 522P. As a result, the vein ends 521P, 522P are likely to be set at more appropriate positions.

[0094] In addition, in FIG. 11 to FIG. 13, the arteriovenous crossing detection process (S9) is performed for one of a pair of veins extending in different directions from a pair of contours of an artery. However, in reality, the CPU 3 performs the arteriovenous crossing detection process for each of a pair of veins extending in different directions from a pair of contours of an artery. That is, the CPU 3 sets each of a pair of regions that spread along the veins in different directions from the pair of contours of the artery in the crossing portion 70 of the fundus image 30 as an attention area AL, and performs the arteriovenous crossing detection process separately for each of the pair of attention areas AL. The arteriovenous crossing may occur only in one of the pair of veins extending in different directions from the artery. Therefore, the detection process for the presence or absence of the arteriovenous crossing is performed separately for the pair of attention areas AL, so that the arteriovenous crossing can be appropriately detected even if the arteriovenous crossing occurs only in one of the attention areas.

[0095] The results of verifying the usefulness of the technology exemplified in the above embodiment will be described with reference to FIG. 14. In this verification, first, an ophthalmologist visually judged the presence or absence of the arteriovenous crossing phenomenon for each of 100 fundus images 30. As a result, 11 images were judged by the ophthalmologist as "present" and 89 images were judged as "absent". Next, the fundus image processing device 1 was made to detect (judge) the presence or absence of the arteriovenous crossing phenomenon for each of the 100 fundus images 30 identical to the images judged by the ophthalmologist. As a result, 44 images were judged by the fundus image processing device 1 as "present" and 56 images were judged by the fundus image processing device 1 as "absent". Here, the number of images judged by the fundus image processing device 1 as "absent" despite the ophthalmologist's judgment of "present" was zero. Therefore, according to the technology exemplified in the above embodiment, the possibility of overlooking the arteriovenous crossing phenomenon is low. Furthermore, the ophthalmologist re-evaluated 33 images that the fundus image processing device 1 had determined to have the phenomenon even though the ophthalmologist had determined that the phenomenon did not exist. As a result, the 33 images included three images that were determined to have the phenomenon after the re-evaluation. As described above, the arteriovenous crossing phenomenon can be detected with high accuracy according to the technology exemplified in the above embodiment.

[0096] The techniques disclosed in the above embodiments are merely examples. Therefore, the techniques exemplified in the above embodiments can be modified. It is also possible to execute only a part of the processes exemplified in the above embodiments. For example, when a part of the fundus image 30 including the optic nerve is included in the target for detecting the arteriovenous intersection, the process (S3) for identifying the position of the optic nerve may be omitted.

[0097] The process of acquiring a fundus image in S1 of FIG. 3 is an example of a "fundus image acquiring step". The process of acquiring a blood vessel image in S2 of FIG. 3 is an example of a "blood vessel image acquiring step". The process of detecting a crossing in S6 of FIG. 3 is an example of a "crossing detection step". The process of acquiring vein pixel information in S7 of FIG. 3 is an example of a "vein pixel information acquiring step". The process of detecting the presence or absence of an arteriovenous crossing phenomenon in S9 of FIG. 3 is an example of a "crossing phenomenon determination step". The process of thinning a blood vessel image in S4 of FIG. 3 is an example of a "thinning step". The process of identifying a blood vessel group in S5 of FIG. 3 is an example of a "blood vessel group identification process". The process of acquiring external pixel information in S8 of FIG. 3 is an example of an "external pixel information acquiring step". [Explanation of symbols]

[0098] 1 Fundus image processing device 3 CPU 4 Storage device 11 Fundus imaging device 30 Fundus image 40 Vascular Images 41 Thinned Image 51A Artery 51B Vein 52A Thinning artery 52B Thinning veins 53 Vascular Group 55 Pixel of interest 70 Cross section AR(AR1,AR2) Area of ​​interest BL reference straight line SL Search Line

Claims

1. A fundus image processing device that processes a fundus image of a subject's eye, The control unit of the fundus image processing device includes: a fundus image acquiring step of acquiring a fundus image including blood vessels of the fundus of the subject's eye, the fundus image being captured by a fundus image capturing device; a blood vessel image acquiring step of acquiring blood vessel images of an artery and a vein included in the acquired fundus image; a crossing portion detection step of detecting at least one of a crossing portion where an artery and a vein cross in the fundus image by processing the blood vessel image; a vein pixel information acquiring step of acquiring vein pixel information, which is pixel information of a portion of the vein appearing in the fundus image that is spaced from the intersection; a crossing phenomenon determination step of determining whether or not an arteriovenous crossing phenomenon occurs at the crossing portion of the fundus image by determining, as a region of interest, a region extending from a contour of an artery along a vein, and comparing pixel information within the region of interest in the fundus image with vein pixel information; A fundus image processing apparatus comprising:

2. The fundus image processing device according to claim 1 , The control unit is further performing a thinning step on the blood vessel image to thin at least a width of a vein; The fundus image processing apparatus is characterized in that, in the crossing detection step, the crossing is detected by processing a thinned image in which at least the width of the vein is thinned.

3. The fundus image processing device according to claim 1 , The control unit is A fundus image processing device characterized in that it further performs a blood vessel group identification process to detect branching points where blood vessels branch off for at least the veins in the blood vessel image, and to identify each of a plurality of blood vessel portions branching off from the detected branching points and each connected in a single line as a different blood vessel group.

4. The fundus image processing device according to claim 3, When a blood vessel group is a blood vessel portion that is connected in a single line without branching and has two ends, The control unit, in the crossover detection step, A fundus image processing device characterized by detecting, as the intersection, a pair of ends of a vein that satisfy both the condition that an artery exists between the two ends and the condition that each of the two ends belongs to a different blood vessel group.

5. The fundus image processing device according to claim 1 , The control unit of this fundus image processing device is characterized in that it excludes a ring-shaped area located on the outer periphery of the fundus image from the target area for determining whether or not arteriovenous crossing occurs in the crossing phenomenon determination step.

6. The fundus image processing device according to claim 1 , The control unit of this fundus image processing device is characterized in that it excludes a portion of the area of ​​the fundus image that includes the optic disc that appears in the fundus image from the target area for determining whether or not arteriovenous crossing phenomenon occurs in the crossing phenomenon judgment step.

7. The fundus image processing device according to claim 1 , The control unit of this fundus image processing device is characterized in that it excludes veins in the fundus image whose thickness on the vascular image is below a threshold value from the veins that are to be determined as to whether or not arteriovenous crossing occurs in the crossing phenomenon determination step.

8. The fundus image processing device according to claim 1 , In the crossover phenomenon determination step, the control unit: A fundus image processing device characterized in that, among the intersection portions of the fundus image, each of a pair of regions extending in different directions along a vein from a pair of contours of an artery is set as an area of ​​interest, and whether or not an arteriovenous crossing phenomenon occurs is determined separately for each of the pair of areas of interest.

9. The fundus image processing device according to claim 1 , The control unit is further performing an external pixel information acquiring step of acquiring external pixel information, which is pixel information outside a position determined to be a vein in the blood vessel image, from the fundus image; In the crossing phenomenon judgment step, pixel information within the area of ​​interest in the fundus image is compared with the vein pixel information and the external pixel information to judge whether or not an arteriovenous crossing phenomenon is occurring at the crossing point.

10. The fundus image processing device according to claim 9, In the crossover phenomenon determination step, the control unit: determining whether pixel information within the region of interest in the fundus image is closer to the vein pixel information or the external pixel information; A fundus image processing apparatus characterized in that it judges whether or not an arteriovenous crossing phenomenon occurs depending on whether or not the degree to which the pixel information is judged to be close to external pixel information is equal to or greater than a threshold value.

11. The fundus image processing device according to claim 9, In the external pixel information acquisition step, the control unit detecting an end of a portion of a vein that appears in the fundus image as if it is divided by crossing an artery at the crossing portion; A fundus image processing apparatus characterized in that pixel information of a site adjacent to the outside of the detected end of the vein is obtained as the external pixel information.

12. The fundus image processing device according to claim 1 , Each pixel of the fundus image is represented by three primary colors of RGB, A fundus image processing device characterized in that, in the crossing phenomenon judgment step, the control unit judges whether or not an arteriovenous crossing phenomenon is occurring at the crossing point based on pixel information excluding the B value among the R value, G value, and B value.

13. The fundus image processing device according to claim 1 , In the crossover phenomenon determination step, the control unit: A fundus image processing device characterized in that a search line, which has a shape that follows the contour of the artery at the crossing and has a length set to be less than the width of the vein at the crossing, is moved from the contour of the artery in a direction along the vein, so that the area through which the search line passes is included in the area of ​​interest.

14. The fundus image processing device according to claim 13, In the crossover phenomenon determination step, the control unit: a reference straight line connecting end portions of a pair of veins extending in different directions from the crossing portion in the fundus image, the end portions being on the crossing portion side; A fundus image processing device characterized in that the area through which the search line passes when the search line is moved from the contour line of the artery to the end along the reference straight line is included in the area of ​​interest.

15. The fundus image processing device according to claim 13, The control unit is further performing a thinning step on the blood vessel image to thin at least a width of a vein; A fundus image processing device characterized in that in the crossing phenomenon judgment step, the ends of a pair of veins extending in different directions from the crossing point are detected based on the positions of each end point of the pair of veins thinned in the thinning step.

16. The fundus image processing device according to claim 15, In the crossover phenomenon determination step, the control unit:

4. A fundus image processing apparatus, comprising: a position spaced a predetermined distance along the thinned vein from an end point of the thinned vein in the thinning step, as the position of the end of the vein.

17. A fundus image processing program executed by a fundus image processing device for processing a fundus image of a subject's eye, The fundus image processing program is executed by a control unit of the fundus image processing device, a fundus image acquiring step of acquiring a fundus image including blood vessels of the fundus of the subject's eye, the fundus image being captured by a fundus image capturing device; a blood vessel image acquiring step of acquiring blood vessel images of an artery and a vein included in the acquired fundus image; a crossing portion detection step of detecting at least one of a crossing portion where an artery and a vein cross in the fundus image by processing the blood vessel image; a vein pixel information acquiring step of acquiring vein pixel information, which is pixel information of a portion of the vein appearing in the fundus image that is spaced from the intersection; a crossing phenomenon determination step of determining whether or not an arteriovenous crossing phenomenon occurs at the crossing portion of the fundus image by determining, as a region of interest, a region extending from a contour of an artery along a vein, and comparing pixel information within the region of interest in the fundus image with vein pixel information; A fundus image processing program that causes the fundus image processing device to execute the above steps.

Images