Image processing device, image processing program, and image processing method

Abstract

To detect an intersection between an artery and a vein from a photographed image obtained by imaging a blood vessel.SOLUTION: An image processing device 1 according to the present invention specifies an intersection of a blood vessel on the basis of a photographed image i1. An image processing device comprises: a storage unit 12 which stores a thinned image i3; a detection unit 133 which detects a plurality of detection points corresponding to each of an intersection, a branch point and an end point of the blood vessel thinned on the basis of the thinned image; a set information specification unit 134 which specifies set information on the basis of the thinned image; a blood vessel graph generation unit 135 which generates a blood vessel graph i4 on the basis of the set information; a connection information acquisition unit 136 which acquires connection information on the basis of the blood vessel graph; and an intersection specification unit 137 which specifies a detection point corresponding to the intersection on the basis of the connection information. The intersection specification unit extracts two branching detection points as a detection point pair on the basis of the connection information and specifies the branching detection point constituting the detection point pair satisfying a predetermined prescribed condition as a detection point corresponding to the intersection.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing program, and an image processing method. [Background technology]

[0002] For early detection of vascular disorders in the brain, medical devices that visualize the inside of the human body, such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging), are used. However, these medical devices are expensive. Therefore, the burden of capital investment is large, and only a limited number of hospitals have these medical devices. In addition, medical expenses are high, so the medical expense burden on patients is also large.

[0003] The brain is the same organ as the eye in terms of development. Therefore, the state of blood vessels in the brain can be inferred from the state of retinal blood vessels in the fundus. Therefore, fundus examinations, which do not require the aforementioned medical equipment, have been used to diagnose the risk of vascular disorders in the brain. In recent years, fundus examinations have been incorporated into internal medicine examinations due to the revision of the third stage of specific health checkups. However, fundus examinations are largely dependent on the experience and skill of the assessing physician. Therefore, it is difficult for non-specialist internal medicine physicians to perform accurate fundus examinations. Therefore, there is a demand for the automation of fundus examinations. To automate fundus examinations, image processing technology is required to accurately extract blood vessels from images of the fundus.

[0004] Fundus examinations can be used not only to prevent and detect eye diseases early, but also to estimate the progression of arteriosclerosis in blood vessels in the brain. Arteriosclerosis does not progress at a constant rate, but at an accelerated rate. Therefore, early treatment of arteriosclerosis can be achieved by observing the degree of arteriosclerosis over time and identifying patients whose arteriosclerosis is accelerating.

[0005] Among the indices for evaluating the progression of arteriosclerosis, the ratio of venous diameters at the intersection of an artery and a vein (diameter of the part influenced by an artery (V2) / diameter of the part not influenced by an artery (V1): V2 / V1 ratio) is known as an index that is not affected by vascular pulsation (see, for example, Patent Document 1).

[0006] In order to automate fundus examinations using the V2 / V1 ratio, a technology for automatically detecting the intersections of retinal arteries and veins using image processing is required. Here, in a known method using a Convolutional Neural Network (CNN), a large amount of training data is required for training the CNN. However, it is difficult to prepare a large amount of training data that captures images of the intersections of retinal blood vessels, which are also medical data.

[0007] Techniques for automatically detecting crossing points of blood vessels as an alternative to CNN have been proposed (see, for example, Patent Document 2).

[0008] The technology disclosed in Patent Document 2 generates a thinned blood vessel image in which blood vessels are thinned from a fundus image, generates a thinned blood vessel image with blood vessel diameter calculated by a blood vessel diameter measurement means, and extracts blood vessel intersections from the thinned blood vessel image. In extracting these intersections, for the thinned branching point, the angle and other branching points / intersections / end points connected thereto are measured, pairs of branching points with short distances between them are detected, and a branching point pair that satisfies a predetermined condition based on a predetermined judgment criterion is regarded as an intersection. As a result, this technology can detect intersections that appear as two branching point pairs by thinning blood vessels.

[0009] However, this technology requires many items as criteria for judgment, and the specific calculation method for each item is unclear. In addition, the comparison target for the specified conditions to be satisfied is unclear. Therefore, this technology lacks objectivity and cannot be used as it is to detect the intersection of blood vessels. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2008-079682 A [Patent Document 2] JP 2009-189586 A Summary of the Invention [Problem to be solved by the invention]

[0011] An object of the present invention is to detect an intersection point between an artery and a vein from a captured image of blood vessels. [Means for solving the problem]

[0012] The image processing device according to the present invention is an image processing device that identifies vascular intersections based on a captured image in which blood vessels are captured, and includes a storage unit that stores a thinned image of blood vessels included in the captured image, a detection unit that detects a plurality of detection points corresponding to any of intersections, branching points, and end points of the thinned blood vessels based on the thinned image, a group information identification unit that identifies, based on the thinned image, group information indicating a pair of detection points connected by a thinned blood vessel out of the plurality of detected detection points, and a blood vessel graph generation unit that generates a blood vessel graph in which each of the two detection points constituting a pair is connected by a line segment based on the identified group information. The system comprises a connection information acquisition unit that acquires connection information for each of the detected detection points based on the vascular graph, and a crossing point identification unit that identifies a detection point that corresponds to a crossing point from the detected detection points based on the connection information, wherein the crossing point identification unit extracts a plurality of branch detection points that are connected to three line segments from the detected detection points based on the connection information, extracts two branch detection points that are connected to a common line segment from the extracted branch detection points based on the connection information as a detection point pair, and identifies the branch detection points that constitute the detection point pair and that satisfy a predetermined condition as the detection point that corresponds to the crossing point. Effect of the Invention

[0013] According to the present invention, it is possible to detect the intersection of an artery and a vein from a captured image of blood vessels. [Brief description of the drawings]

[0014] [Figure 1] 1 is a network connection diagram showing an embodiment of an image processing apparatus according to the present invention; [Diagram 2] 1 is a functional block diagram showing an embodiment of an image processing apparatus according to the present invention; [Diagram 3] 3 is a schematic diagram showing an example of a captured image processed by the image processing device of FIG. 2; [Figure 4] 1 is a flowchart illustrating an embodiment of an image processing method according to the present invention. [Diagram 5] 5 is a flowchart of a blood vessel graph generation process included in the image processing method of FIG. 4. [Figure 6] 6 is a schematic diagram showing an example of an extracted image generated in the blood vessel graph generation process of FIG. 5. [Figure 7] 6 is a schematic diagram showing an example of a thinned image generated in the blood vessel graph generation process of FIG. 5. [Figure 8] 3 is a schematic diagram showing an example of a filter stored in a storage unit included in the image processing device of FIG. 2; [Figure 9] 9 is a schematic diagram showing an example of a state in which each filter pattern in FIG. 8 is applied to the thinned image in FIG. 7. [Figure 10] 6 is a flowchart of a group identification process included in the vascular graph generation process of FIG. 5. [Figure 11] 11 is a schematic diagram showing an example of a blood vessel search in the group identification process of FIG. 10. [Figure 12] FIG. 6 is a schematic diagram showing an example of a blood vessel graph generated by the blood vessel graph generation process of FIG. 5. [Figure 13] 5 is a flowchart of a link information acquisition process included in the image processing method of FIG. 4. [Figure 14] 14 is a schematic diagram showing an example of detection points from which link information is acquired in the link information acquisition process of FIG. 13. FIG. [Figure 15]14 is a schematic diagram showing a method of calculating an angle between line segments in the connection information acquisition process of FIG. 13. [Figure 16] 14 is a schematic diagram showing a method of calculating an aperture in the connection information acquisition process of FIG. 13. [Figure 17] 5 is a flowchart of a crossing point detection process included in the image processing method of FIG. 4. [Figure 18] 18 is a schematic diagram showing detection points that can be detected as detection points corresponding to crossing points in the crossing point detection process of FIG. 17; FIG. [Figure 19] 5 is a schematic diagram showing the success rate and the failure rate when blood vessel crossings are detected from an actual fundus image by the image processing method of FIG. 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Below, with reference to the drawings, we will explain embodiments of an image processing device (hereinafter referred to as "the device"), an image processing program (hereinafter referred to as "the program"), and an image processing method (hereinafter referred to as "the method") according to the present invention.

[0016] The present invention detects a plurality of detection points based on a thinned image in which blood vessels included in an image in which blood vessels are captured are thinned, and specifies group information of the detection points. The present invention also generates a blood vessel graph based on the group information, obtains connection information for each of the detection points based on the blood vessel graph, and detects detection points corresponding to intersections based on the connection information.

[0017] In the following embodiment, the present invention will be described using a fundus image of an artery and a vein as an example of a captured image. The captured image includes a portion where an artery and a vein cross each other (hereinafter referred to as a "crossing point").

[0018] The "captured image" includes, for example, a fundus image captured by a fundus camera, an X-ray image captured by an X-ray imaging device, a CT image captured by a CT device, and also an image captured by a normal optical camera. In the present invention, the captured image includes at least blood vessels as a subject. As described above, in the present embodiment, the captured image is a fundus image.

[0019] The term "blood vessels" includes not only human blood vessels but also blood vessels of animals other than humans. In this embodiment, the blood vessels are blood vessels in the fundus of a human eye.

[0020] A "detection point" is a point (pixel) that corresponds to any of the intersections, branching points, and end points of thinned blood vessels. Here, an intersection means a portion where an artery and a vein intersect in a captured image, a branching point means a portion where one blood vessel splits into two blood vessels (branch vessels) in a captured image, and an end point means an end of a blood vessel in a captured image.

[0021] "Group information" is information indicating a pair of two detection points that are connected by a thinned blood vessel among a plurality of detection points. In other words, group information is information indicating a detection point that is connected to another detection point among a plurality of detection points. Group information includes pixel information (coordinates, number of pixels) of blood vessels that are connected to the two detection points that make up the pair.

[0022] A "vascular graph" is a graph of blood vessels captured in a captured image, with each detection point as a node and each line segment connecting multiple detection points as an edge. In other words, a blood vessel graph is composed of multiple detection points and multiple line segments connecting the detection points.

[0023] "Connection information" is information related to objects (other detection points, line segments, blood vessels) connected to a detection point. The connection information includes, for example, the number of line segments connected to the detection point (hereinafter also referred to as "order"), the angle between the line segments connected to the detection point, other connection points connected to the detection point via line segments, the length of the blood vessel connected to the detection point (blood vessel connected to two detection points), and the caliber of the blood vessel connected to the detection point.

[0024] ●Image processing device● First, an embodiment of the present device will be described.

[0025] FIG. 1 is a network connection diagram showing an embodiment of the present device. The figure shows that the present device 1 is connected to an imaging device 2 via a network (communication line) N that uses a wired communication method or a wireless communication method.

[0026] The device 1 identifies the intersections of blood vessels based on a captured image i1 of the blood vessels. The specific configuration and operation of the device 1 and the captured image i1 will be described later.

[0027] The imaging device 2 captures an image i1. The imaging device 2 is, for example, a fundus camera.

[0028] The network N is, for example, a communication network such as the Internet, a mobile communication network, a local area network (LAN), a wide area network (WAN), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] ●Configuration of image processing device FIG. 2 is a functional block diagram showing an embodiment of the present device 1. As shown in FIG.

[0030] The present device 1 is realized, for example, by a personal computer. The present program runs on the present device 1, and the present program cooperates with the hardware resources of the present device 1 to realize the present method.

[0031] Here, by causing a computer (not shown) to execute this program, the program can cause the computer to function in the same manner as the device 1, and to execute the method.

[0032] The device 1 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0033] The communication unit 11 acquires the captured image i1 from the imaging device 2 via the network N. The communication unit 11 is configured with, for example, a communication module and an antenna. The captured image i1 is stored in the storage unit 12.

[0034] The storage unit 12 stores information necessary for the device 1 to execute the method described below. The storage unit 12 is configured, for example, with a recording device such as a hard disk drive (HDD) or a solid state drive (SSD) provided in the device 1, a volatile memory such as a random access memory (RAM), and / or a portable storage medium such as a flash memory.

[0035] The control unit 13 controls the operation of the entire device 1 and executes the method described below. The control unit 13 is composed of, for example, a CPU (Central Processing Unit) included in the device 1, a volatile memory such as a RAM that functions as a working area for the CPU, and a non-volatile memory such as a ROM (Read Only Memory) that stores various information such as the program. The control unit 13 includes an extracted image generating unit 131, a thinned image generating unit 132, a detection unit 133, a group information identifying unit 134, a blood vessel graph generating unit 135, a connection information acquiring unit 136, and an intersection identifying unit 137.

[0036] The extraction image generating section 131 generates an extraction image i2 based on the captured image i1. A specific operation of the extraction image generating section 131 will be described later.

[0037] The "extracted image i2" is an image obtained by extracting a blood vessel region included in the captured image i1.

[0038] The thin line image generating unit 132 generates a thin line image i3 based on the extracted image i2. The specific operation of the thin line image generating unit 132 will be described later.

[0039] The thinned image i3 is an image in which the blood vessel region included in the extracted image i2 has been thinned to a width of one pixel. In other words, the thinned image i3 is an image in which the blood vessels included in the captured image i1 have been thinned.

[0040] The detection unit 133 detects a plurality of detection points corresponding to any of the intersections, branching points, and end points of the thinned blood vessels based on the thinned image i3. The specific operation of the detection unit 133 will be described later.

[0041] The group information identifying unit 134 identifies group information indicating a pair of two detection points connected by a thinned blood vessel among the detected detection points based on the thinned image i3. A specific operation of the group information identifying unit 134 will be described later.

[0042] Based on the identified pair information, the blood vessel graph generating unit 135 generates a blood vessel graph i4 in which each of the two detection points constituting a pair is connected by a line segment. The specific operation of the blood vessel graph generating unit 135 will be described later.

[0043] The connection information acquisition unit 136 acquires connection information for each of the detected detection points based on the extracted image i2, the thinned image i3, and the blood vessel graph i4. The specific operation of the connection information acquisition unit 136 will be described later.

[0044] The intersection point identification unit 137 identifies a detection point that corresponds to the intersection point among the detected detection points based on the connection information. A specific operation of the intersection point identification unit 137 will be described later.

[0045] In addition, the extracted image generation unit, thinned image generation unit, detection unit, group information identification unit, vascular graph generation unit, connection information acquisition unit, and intersection point identification unit in the present invention may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to or instead of a CPU.

[0046] In addition, the extracted image generation unit, thinned image generation unit, detection unit, group information identification unit, vascular graph generation unit, connection information acquisition unit, and intersection point identification unit in the present invention may be configured using common hardware (such as a CPU) or may be configured using separate hardware.

[0047] ●Image processing method (operation of image processing device)● Next, the operation of the present device 1, that is, an embodiment of the present method executed by the present device 1, will be described with reference to FIG.

[0048] In the following description, it is assumed that the device 1 acquires a captured image i1 from the imaging device 2 via the network N and stores it in the storage unit 12. It is also assumed that the captured image i1 used in the following description captures an image of an intersection point where a part of a vein and an artery cross each other.

[0049] FIG. 3 is a schematic diagram showing an example of a captured image i1. The figure shows that part of an artery branches into two, that one of the branched arteries has an end point, and that parts of the artery and vein cross each other, resulting in part of the vein being obscured by part of the artery and not being imaged.

[0050] FIG. 4 is a flow chart illustrating an embodiment of the method.

[0051] This method includes a blood vessel graph generating process (S1), a connection information acquiring process (S2), and an intersection point detecting process (S3).

[0052] - Blood vessel graph generation processing First, the device 1 executes a blood vessel graph generation process (S1).

[0053] FIG. 5 is a flowchart of the blood vessel graph generation process (S1).

[0054] The "blood vessel graph generation process (S1)" is a process for generating a blood vessel graph i4 from a captured image i1.

[0055] First, the extracted image generating unit 131 reads out the captured image i1 from the storage unit 12, and performs known image processing (for example, processing using a CNN (Convolutional Neural Network) using learning, or processing using a BTH (Black-Top-Hat) transform, which is a type of morphological operation) on the captured image i1 to extract the blood vessel area included in the captured image i1 and generate an extracted image i2 (S101). That is, for example, the extracted image generating unit 131 binarizes the blood vessel area included in the captured image i1 into white and the area other than the blood vessel area (background area) into black, thereby generating an extracted image i2 in which the captured image i1 has been binarized. The extracted image i2 is stored in the storage unit 12.

[0056] FIG. 6 is a schematic diagram showing an example of the extracted image i2. This figure shows that the blood vessel region extracted from the captured image i1 shown in FIG. 1 is represented in white (pixel value "1"), and the background region is represented in black (pixel value "0").

[0057] Return to Figure 5. Next, the thinned image generating unit 132 performs a known thinning process on the extracted image i2 to thin the blood vessel region included in the extracted image i2 to a width of one pixel, thereby generating a thinned image i3 (S102). The thinned image i3 is stored in the storage unit 12.

[0058] The “known thinning process” is an image process for thinning white areas of a binarized image, and is, for example, the process described in “Zhang TY, and Ching Y. Suen, “A fast parallel algorithm for thinning digital patterns”, Communications of the ACM, (US), 1984, Vol. 27, No. 3, pp. 236-239”.

[0059] FIG. 7 is a schematic diagram showing an example of a thinned image i3. This figure shows that the blood vessel region of the extracted image i2 shown in FIG. 2 has been thinned to a width of one pixel.

[0060] Return to Figure 5. Next, the detection unit 133 applies each pattern of the filter F to each pixel of the thinned image i3 to identify pixels that are detection points (S103). That is, the detection unit 133 executes a filter process using the filter F on the thinned image i3.

[0061] The "filter F" is a filter that the detection unit 133 applies to the thinned image i3 in order to identify the intersections, branching points, end points, and passing points of blood vessels from each pixel of the thinned image i3. The filter F includes a plurality of patterns corresponding to the intersections, branching points, end points, and passing points of blood vessels. The filter F is stored in the storage unit 12 in advance.

[0062] FIG. 8 is a schematic diagram showing an example of the filter F stored in the storage unit 12. As shown in FIG. This figure shows that filter F includes multiple patterns made up of 3x3 pixels. This figure shows that filter F includes a first group in which a different pattern appears when rotated, and a second group in which no other pattern appears when rotated. This figure shows that each pixel in each pattern is binarized (white: pixel value "1", black: pixel value "0").

[0063] Specifically, the detection unit 133 selects one pixel as a target pixel from among the pixels constituting the thinned image i3, and refers to a detection range of 3×3 pixels including eight pixels surrounding the target pixel. Next, the detection unit 133 superimposes each pattern on the detection range to determine whether or not there is a pattern that matches the detection range. When there is a pattern that matches the detection range, the detection unit 133 specifies the pixel at the center of the detection range (i.e., the target pixel) as a pixel that is a detection point. Next, the detection unit 133 calculates the inner product of the pixel value ("0" or "1") of the pixel in the detection range and the pixel value ("0" or "1") of the pixel constituting the pattern that matches the detection range. Next, when the inner product is "2", the detection unit 133 specifies the target pixel as an end point, when the inner product is "4" or more, the detection unit 133 specifies the target pixel as a branch point (including a crossing point), and when the inner product is "3", the detection unit 133 specifies the target pixel as a pass point. At this point, no distinction is made between branch points and intersection points. Coordinate information of the identified detection points (end points, branch points) is stored in the storage unit 12, for example, in association with the thinned image i3.

[0064] Note that the detection section in the present invention may specify the target pixel as either an end point or a branch point (including an intersection point) based on the sum of the number of pixels with a pixel value of "1" in the detection range, instead of the inner product.

[0065] FIG. 9 is a schematic diagram showing an example of a state in which each pattern of the filter F is applied to each pixel of the thinned image i3. In the figure, “a” indicates that the inner product is “4 (branching point)”, “b” indicates that the inner product is “3 (passing point)”, “c” indicates that the inner product is “2 (end point)”, and “d” indicates that the inner product is “5 (crossing point included in the branching point)”.

[0066] Return to Figure 5. Next, the group information identifying unit 134 executes a group identifying process (S104).

[0067] FIG. 10 is a flowchart of the group identification process (S104). FIG. 11 is a schematic diagram showing an example of blood vessel search in the group identification process (S104). 11, for ease of explanation, the central pixel is labeled "x1", and the eight pixels surrounding the central pixel are labeled "x2" through "x9". In addition, some pixels in a second region described below are labeled "x10" through "x12". In the following explanation, the pixels surrounding the central pixel are the pixels adjacent to the central pixel diagonally above, below, left, and right, and the pixels adjacent to the central pixel are the pixels adjacent to the central pixel above, below, left, and right.

[0068] The "pair identification process (S104)" is a process for identifying a pair of two detection points connected by a thinned blood vessel among the detected detection points, by searching for thinned blood vessels connected to each detection point based on the thinned image i3 and the coordinate information of the detection points associated with the thinned image i3 from the storage unit 12. In other words, the pair identification process (S104) identifies another detection point connected to a certain detection point via a blood vessel.

[0069] First, the group information identification unit 134 selects one of the detected detection points as a search starting point (S104-1). Next, the group information identification unit 134 recognizes a 3×3 pixel region with the detection point as a first center pixel as a first region (S104-2). Next, the group information identification unit 134 references eight pixels surrounding the first center pixel in the first region as reference pixels (S104-3). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-4). Here, the group information identification unit 134 performs the determination in the order of, for example, "x2" to "x9", and detects the pixel that is first determined to correspond to a specific pixel as a specific pixel in the first region (S104-5).

[0070] A "specific pixel" is a pixel that indicates a thinned blood vessel. A "non-specific pixel" is a pixel that indicates something other than a thinned blood vessel (i.e., the background).

[0071] Next, the group information identification unit 134 recognizes a 3×3 pixel region with the detected specific pixel as the second center pixel as the second region (S104-6). Next, the group information identification unit 134 references, as reference pixels, five pixels, excluding the first center pixel and the pixel adjacent to the first center pixel, among the surrounding pixels surrounding the second center pixel in the second region (S104-7). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-8). Here, the group information identification unit 134 performs the determination in the same order as in the process (S104-5), and detects the pixel that is first determined to be a pixel corresponding to a specific pixel as a specific pixel in the second region (S104-9). Next, the group information identification unit 134 determines whether the detected specific pixel is a detection point other than the detection point that is the starting point (S104-10).

[0072] When the detected specific pixel is not another detection point ("N" in S104-10), the group information identification unit 134 recognizes a 3×3 pixel region with the detected specific pixel as a third center pixel as a third region (S104-11). Next, the group information identification unit 134 references five pixels, excluding the second center pixel and the pixel adjacent to the second center pixel, among the eight pixels surrounding the third center pixel in the third region as reference pixels (S104-12). Next, the group information identification unit 134 determines whether each of the reference pixels corresponds to a specific pixel or a non-specific pixel (S104-13). Here, the group information identification unit 134 performs the determination in the same order as in the process (S104-5), and detects the pixel that is first determined to be a pixel corresponding to a specific pixel as a specific pixel in the third region (S104-14). Next, the group information identification unit 134 determines whether the detected specific pixel is another detection point (S104-15). When the detected specific pixel is not another detection point ("N" in S104-15), the pair identification process (S104) returns to process (S104-6). On the other hand, when the detected specific pixel is another detection point ("Y" in S104-10, 15), the pair information identification unit 134 identifies the pair information (information indicating the coordinate information of the two detection points and the number of specific pixels between the two detection points (hereinafter referred to as "number of pixels between detection points")) of the detection point that is the starting point and the other detection point that is the last detected specific pixel as a pair of detection points (S104-16). The pair information of the detection points is stored in the storage unit 12, for example, in association with the number of pixels between the detection points.

[0073] Return to Figure 5. Next, the blood vessel graph generating unit 135 generates a blood vessel graph i4 by connecting two detection points constituting each pair with a line segment based on the pair information (S105). At this time, the blood vessel graph generating unit 135 assigns an identification number to each detection point based on, for example, a predetermined condition in order to identify each detection point. The blood vessel graph i4 is stored in the storage unit 12 in association with, for example, the pair information and the number of pixels between the detection points.

[0074] FIG. 12 is a schematic diagram showing an example of a blood vessel graph i4. In the figure, the numbers given to each detection point are identification numbers of the respective detection points.

[0075] In this way, the device 1 executes the blood vessel graph generation process (S1) on the captured image i1 to generate a blood vessel graph i4 that represents blood vessels using points (detection points) and straight lines (line segments). By representing blood vessels using a blood vessel graph, the device 1 can easily obtain connection information, which will be described later.

[0076] ● Consolidated information acquisition process Next, the device 1 executes a link information acquisition process (S2: see FIG. 4).

[0077] FIG. 13 is a flowchart of the link information acquisition process (S2).

[0078] The "connection information acquisition process (S2)" is a process for acquiring connection information based on the extracted image i2, the thinned image i3, and the blood vessel graph i4.

[0079] First, the connection information acquisition unit 136 acquires, based on the vascular graph i4, the detection points connected to each detection point, the number (order) of detection points (i.e., line segments) connected to each detection point, and the angles between the line segments connected to each detection point as connection information (S201).

[0080] FIG. 14 is a schematic diagram showing an example of detection points from which link information is acquired. The figure shows an enlarged portion of the blood vessel graph i4. The figure shows that the detection points connected to the detection point "10" are "11", "14", "25", and "30", and the degree of the detection point "10" is "4".

[0081] FIG. 15 is a schematic diagram showing a method for calculating the angle between line segments. The figure shows that when an arbitrary detection point is "P", a detection point connected to the detection point "P" by a line segment "a" is "Q", and a detection point connected to the detection point "P" by a line segment "b" is "R", the angle "θ" between the line segment "ab" can be calculated using the following formula (1).

[0082] θ=cos -1 (a·b / |a|·|b|) (1)

[0083] Return to Figure 13. Next, the connection information acquisition unit 136 acquires the length of the blood vessel connecting the two detection points (hereinafter referred to as "blood vessel length") as connection information based on the number of pixels between the detection points stored in association with the blood vessel graph i4 (S202). Here, the number of pixels between the detection points is the number of specific pixels that exist between the two detection points in the thinned image i3. In other words, the number of pixels between the detection points is the blood vessel length. The number of pixels between the detection points includes the detection points.

[0084] The number of pixels between detection points does not have to include the detection points.

[0085] Next, the connection information acquisition unit 136 calculates the diameter of the blood vessel connecting the two detection points as connection information based on the extracted image i2 and the thinned image i3 (S203).

[0086] FIG. 16 is a schematic diagram showing a method for calculating the aperture. In the figure, detection points are indicated by "●".

[0087] Specifically, the connection information acquisition unit 136 extracts an image (hereinafter referred to as a "selected blood vessel image") including blood vessels (hereinafter referred to as "selected blood vessels") connected only to two arbitrarily selected detection points from the thinned image i3 (S203-1). Next, the connection information acquisition unit 136 performs a known expansion process on the selected blood vessel image to generate an expanded blood vessel image in which the selected blood vessel is expanded (S203-2). Next, the connection information acquisition unit 136 performs a mask process on the expanded blood vessel image using a region of the extracted image i2 corresponding to the expanded blood vessel image as a mask to define a pseudo blood vessel region (S203-3). That is, for example, the connection information acquisition unit 136 processes pixels that correspond to blood vessels in the expanded blood vessel image and that protrude from the masked extracted image i2 as a background. Next, the connection information acquisition unit 136 calculates the blood vessel area (the total number of pixels showing blood vessels) of the pseudo blood vessel region (S203-4). Next, the connection information acquisition unit 136 calculates the caliber of the selected blood vessel using the following formula (2) (S203-5).

[0088] Aperture “w” = (vascular area between two detection points) / (distance between two detection points) (2)

[0089] The acquired connection information is stored in the storage unit 12 in association with the corresponding detection points and line segments.

[0090] Note that the process (S201) may be executed after the process (S202) and the process (S203), or may be executed in parallel with the process (S202) and the process (S203).

[0091] ●Crossing point detection process Next, the device 1 executes a crossing point detection process (S3: see FIG. 4).

[0092] FIG. 17 is a flowchart of the crossing point detection process (S3). Fig. 18 is a schematic diagram showing detection points that may be detected as detection points corresponding to crossing points in the crossing point detection process (S3). In Fig. 18, blood vessels are shown with dashed lines, line segments are shown with solid lines, and detection points are shown with "◯".

[0093] The "intersection detection process (S3)" is a process for detecting a detection point corresponding to an intersection among a plurality of detection points. Most of the branching points of blood vessels are three-way intersections where one blood vessel branches into two blood vessels. Therefore, in the blood vessel graph i4 based on the thinned blood vessels, a detection point with an order of "4" can be regarded as a detection point corresponding to a point where two blood vessels (arteries, veins) intersect (vascular intersection). On the other hand, a detection point with an order of "3" can be regarded as a detection point corresponding to a blood vessel branching point. However, an intersection may appear as two adjacent branching points. In other words, a detection point with an order of "3" may correspond not only to a branching point but also to an intersection. The intersection detection process (S3) is a process for determining whether such a detection point with an order of "3" corresponds to a branching point or to an intersection.

[0094] First, the intersection identification unit 137 extracts a detection point having an order of "4" from among multiple detection points based on the vascular graph i4 and the connection information (S301), and identifies the detection point having an order of "4" as a detection point corresponding to a intersection (S302).

[0095] Next, the intersection point identification unit 137 extracts, from among the multiple detection points, a detection point with an order of "3" as a branch extraction point based on the blood vessel graph i4 and the connection information (S303).

[0096] Next, the intersection point identification unit 137 extracts, from among the branch detection points, two detection points connected to one common line segment (hereinafter referred to as a "common line segment") as a detection point pair (S304). That is, the intersection point identification unit 137 identifies, as a detection point pair, a detection point connected to one end of one common line segment (hereinafter referred to as a "first detection point") and a detection point connected to the other end (hereinafter referred to as a "second detection point"). That is, the detection point pair includes the first detection point and the second detection point connected to the first detection point by the common line segment.

[0097] Next, the intersection point identification unit 137 acquires the length (blood vessel length) "d" of the blood vessel connecting the first detection point and the second detection point and the diameter "w" of the blood vessel connecting the first detection point and the second detection point based on the connection information (S305). Next, the intersection point identification unit 137 determines whether the blood vessel length "d" is smaller than the diameter "w" (S306). The diameter "w" is an example of a first threshold value in the present invention.

[0098] When the blood vessel length “d” is smaller than the caliber “w” (S306: “Y”), the intersection point identification unit 137 determines the angle “θ 1 " and the angle between two of the three line segments connected to the second detection point, excluding the common line segment (hereinafter referred to as the "second angle") "θ 2 Then, the intersection point identification unit 137 acquires the first angle "θ 1 " and the second angle "θ 2 Calculate the absolute value of the difference between the first threshold value and the second threshold value, V th It is determined whether the second threshold value "V th " is, for example, 20° and is stored in advance in the storage unit 12.

[0099] The absolute value is the second threshold "V th " ("Y" in S308), the crossing point identification unit 137 identifies the first detection point and the second detection point that constitute the connection point pair as detection points corresponding to the crossing point (S309). Next, the crossing point identification unit 137 calculates the coordinates of the midpoint between the coordinates of the first detection point and the coordinates of the second detection point, and identifies the coordinates as the coordinates of the detection point pair that corresponds to the crossing point (S310).

[0100] On the other hand, when the blood vessel length “d” is equal to or larger than the caliber “w” (S306 “N”), or when the absolute value is equal to or larger than the second threshold “V th If so ("N" in S308), the crossing point identification unit 137 identifies each of the first detection point and the second detection point constituting the detection point pair as a branch point (S311).

[0101] Next, the crossing point identification unit 137 judges whether or not there are unprocessed detection point pairs (S312). If there are unprocessed detection point pairs ("Y" in S312), the crossing point identification unit 137 executes the process (S304) and subsequent processes. On the other hand, if there are no unprocessed detection point pairs ("N" in S312), the crossing point identification unit 137 stores the coordinates of the detection points (detection point pairs) corresponding to all the identified crossing points in the storage unit 12 in association with information indicating the crossing points (S313).

[0102] In this way, the device 1 determines whether the intersection pair satisfies two conditions, namely, a length condition in which the blood vessel length "d" is compared with a first threshold (caliber "w") and an angle condition in which the absolute value of the difference between the angles is compared with a second threshold "Vth", and specifies the intersection that appears as two adjacent branch points. When two blood vessels cross and overlap, the state in which the two blood vessels are locally parallel at the overlapping portion hardly occurs, and the width of the blood vessel at the overlapping portion of the two blood vessels is approximately the same as the blood vessel caliber. Therefore, by setting the caliber of the blood vessel corresponding to the common line segment as the first threshold, it is possible to determine whether the detection point pair corresponds to the intersection. In addition, by varying the first threshold according to the caliber (thickness) of the blood vessel that forms the intersection, the accuracy of determining whether the detection point pair corresponds to the intersection is improved compared to the case where a fixed value is set as the threshold. Furthermore, when two blood vessels cross, the blood vessel near the intersection can be approximated to a state in which two line segments cross when viewed locally. Therefore, the first angle can be considered to be substantially the same as the second angle. In this method, a blood vessel graph in which blood vessels are replaced with detection points and line segments is used, and the first angle and the second angle can be easily calculated, but a certain degree of difference is likely to occur between the first angle and the second angle. Therefore, by setting the second threshold as an empirical threshold, it is possible to determine whether or not a pair of detection points corresponds to a crossing point.

[0103] FIG. 19 is a schematic diagram showing the success rate and the failure rate when blood vessel intersections are detected from actual fundus images by this method. In the figure, the "number of correct answers" is the number of crossing points visually counted in 40 fundus images. The "correct answer rate" is a percentage of the value obtained by dividing the number of detected crossing points by the number of correct crossing points detected. The "incorrect answer rate" is a percentage of the value obtained by dividing the number of incorrect detections by the total number of detections. The second threshold is 20°. As shown in FIG. 19, the correct answer rate of this method was about 97%, and the incorrect answer rate was about 10%.

[0104] Summary According to the embodiment described above, the present device 1 includes a detection unit 133, a group information specification unit 134, a blood vessel graph generation unit 135, a connection information acquisition unit 136, and a crossing point specification unit 137. The detection unit 133 detects a plurality of detection points based on the thinned image i3. The group information specification unit 134 specifies group information based on the thinned image i3. The blood vessel graph generation unit 135 generates a blood vessel graph i4 based on the specified group information. The connection information acquisition unit 136 acquires connection information of each of the detected detection points based on the blood vessel graph i4. The crossing point specification unit 137 extracts a plurality of branch detection points based on the connection information, extracts two branch detection points that are connected to a common line segment from the extracted plurality of branch detection points as a detection point pair, and specifies a branch detection point that constitutes a detection point pair that satisfies a predetermined condition (length condition, angle condition) as a detection point corresponding to a crossing point. According to this configuration, the present device 1 can easily acquire stable connection information based on a blood vessel graph. As a result, the present device 1 can accurately detect the intersection points between arteries and veins from the captured images.

[0105] Further, according to the embodiment described above, the connection information acquisition unit 136 acquires the angle between the multiple line segments connected to the detection point based on the blood vessel graph i4, acquires the blood vessel length of the blood vessel connecting the two detection points based on the thinned image i3, and acquires the caliber of the blood vessel connecting the two detection points based on the thinned image i3. Normally, when acquiring the blood vessel length based on the captured image i1 and the extracted image i2, it is not easy to specify the start point or end point of the blood vessel from the branch point or intersection point because multiple blood vessels are involved at the branch point or intersection point. However, according to this configuration, since the blood vessel length is acquired by the thinned blood vessel, even if the detection point is a branch point or intersection point, the detection point is indicated by one pixel. Therefore, the present device 1 can easily specify the start point and end point of the blood vessel connecting the two detection points. Also, in calculating the caliber of the blood vessel, the area of ​​the blood vessel at each of the start point and end point of the blood vessel is easy to specify, and the connection information acquisition unit 136 can easily calculate the caliber of the blood vessel based on the area of ​​the blood vessel.

[0106] Furthermore, according to the embodiment described above, the crossing point identification unit 137 identifies the detection point pair as a crossing point when the blood vessel length between the first detection point and the second detection point is smaller than a predetermined first threshold value and the absolute value of the difference between the first angle and the second angle is smaller than a predetermined second threshold value. That is, the crossing point identification unit 137 identifies the detection point pair as a crossing point when the detection point pair satisfies the length condition and the angle condition. According to this configuration, the present device 1 can accurately detect the crossing point between the artery and the vein from the captured image by performing a logical sum of the judgments based on the two threshold values.

[0107] Furthermore, according to the embodiment described above, the first threshold is the diameter of the blood vessel connecting the first detection point and the second detection point. With this configuration, the device 1 can accurately determine whether or not the detection point pair corresponds to the crossing point. In addition, by varying the first threshold according to the diameter (thickness) of the blood vessel forming the crossing point, the accuracy of determining whether or not the detection point pair corresponds to the crossing point is improved compared to the case where a fixed value is used as the threshold.

[0108] Furthermore, according to the embodiment described above, the connection information acquisition unit 136 acquires the caliber as connection information based on the thinned image i3 and the extracted image i2. As described above, this configuration allows the device 1 to easily identify the start and end points of the blood vessel based on the thinned image i3, and to accurately identify the area and width of the blood vessel based on the extracted image i2.

[0109] Furthermore, according to the embodiment described above, the connection information acquisition unit 136 extracts thinned blood vessels that connect two detection points that form a pair from the thinned image i3, and acquires the diameter of the extracted blood vessels by performing a dilation process and a filter process using the extracted image i2 as a filter. With this configuration, the device 1 can easily identify the start point and end point of the blood vessel from which the diameter is to be acquired, and can calculate the area of ​​the blood vessel and the diameter based on the area in an extremely stable manner, since the extracted image, which is an image from which the blood vessel is extracted, is used as a filter.

[0110] Furthermore, according to the embodiment described above, the detection unit 133 detects detection points by executing a filter process on the thinned image using the filter F. According to this configuration, the processing load required for detecting detection points is reduced.

[0111] Furthermore, according to the embodiment described above, the group information identifying unit 134 detects a specific pixel from eight pixels surrounding the first center pixel in the first region. Next, the group information identifying unit 134 identifies group information by repeating a process of detecting a new specific pixel from five pixels excluding the first center pixel and pixels adjacent to the first center pixel among the eight pixels surrounding the second center pixel in the second region, and a process of detecting a new specific pixel from five pixels excluding the second center pixel and pixels adjacent to the second center pixel among the eight pixels surrounding the third center pixel in the third region. According to this configuration, the group information identifying unit 134 can identify group information (thinned blood vessels connecting two detection points) at high speed and with a small processing load.

[0112] In the embodiment described above, the crossing point identification unit 137 identifies a detection point pair that satisfies both the length condition and the angle condition as a detection point corresponding to a crossing point. Alternatively, the crossing point identification unit 137 may identify a detection point pair that satisfies only the length condition as a detection point corresponding to a crossing point.

[0113] In the embodiment described above, the present device 1 includes the extracted image generating unit 131 and the thinned image generating unit 132. Alternatively, the present device may not include the extracted image generation unit, or may not include the extracted image generating unit and the thinned image generating unit. In this case, the present device does not generate the extracted image and the thinned image, but instead obtains the extracted image and the thinned image generated by another device via a network, for example, and stores them in the storage unit.

[0114] Furthermore, in the above-described embodiment, the first threshold is the diameter of the blood vessel corresponding to the common line segment. Alternatively, the first threshold in the present invention may be fixed to a predetermined value.

[0115] Furthermore, the second threshold in the present invention is not limited to 20°. That is, for example, the second threshold may be adjusted and set to an appropriate value by referring to the correct answer rate and the incorrect answer rate.

[0116] Furthermore, the group information specifying section in the present invention may detect the specific pixel by known image processing.

[0117] Furthermore, the storage unit in the present invention may store filter patterns corresponding to the end points.

[0118] Furthermore, the connection information acquiring section in the present invention may specify the blood vessel length based on a blood vessel graph. That is, for example, the connection information acquiring section in the present invention may determine the length of a line segment as the blood vessel length. [Explanation of symbols]

[0119] 1 Image processing device 12 Storage section 133 Detection Unit 134 Group information identification section 135 Blood vessel graph generation unit 136 Consolidated Information Acquisition Department 137 Crossing point identification part

Claims

1. 1. An image processing device that identifies crossing points of blood vessels based on an image of the blood vessels, comprising: a storage unit that stores a thinned image in which the blood vessels included in the captured image are thinned; a detection unit that detects a plurality of detection points corresponding to any of the crossing points, branching points, and end points of the thinned blood vessels based on the thinned image; a group information specifying unit that specifies group information indicating a pair of two of the detection points that are connected by the thinned blood vessel, among the plurality of detection points detected based on the thinned image; a blood vessel graph generating unit that generates a blood vessel graph by connecting each of the two detection points that constitute the pair by a line segment based on the specified pair information; a connectivity information acquisition unit that acquires connectivity information of each of the detected detection points based on the blood vessel graph; a crossing point identification unit that identifies a detection point corresponding to the crossing point among the plurality of detection points detected based on the connection information; and The intersection point identification unit is extracting a plurality of branch detection points connected to three of the line segments from the plurality of detected detection points based on the connection information; extracting, as a detection point pair, two of the branch detection points that are connected to a common line segment from among the extracted branch detection points based on the connection information; identifying the branch detection points constituting the detection point pair as the detection points corresponding to the crossing point based on the connection information of the two branch detection points constituting the detection point pair; 13. An image processing device comprising:

2. The link information acquisition unit acquiring angles between the plurality of line segments connected to the detection point as the connection information based on the blood vessel graph; acquiring, as the connection information, a length of the blood vessel connecting the two detection points based on the thinned image; acquiring, as the connection information, a diameter of the blood vessel connecting the two detection points based on the thinned image; 2. The image processing device according to claim 1.

3. The detection point pair is A first detection point; a second detection point connected to the line segment common to the first detection point; Including, The intersection point identification unit is when the length of the blood vessel connecting the first detection point and the second detection point is smaller than a predetermined first threshold, the detection point pair is identified as the detection point corresponding to the intersection point.

3. The image processing device according to claim 2.

4. The intersection point identification unit, when an absolute value of a difference between the angle between two of the three line segments connected to the first detection point, excluding the common line segment, and the angle between two of the three line segments connected to the second detection point, excluding the common line segment, is smaller than a predetermined second threshold value, the detection point pair is identified as the detection point corresponding to the intersection point.

4. The image processing device according to claim 3.

5. the first threshold value is the diameter of the blood vessel connecting the first detection point and the second detection point; 4. The image processing device according to claim 3.

6. the storage unit stores an extracted image in which the blood vessels included in the captured image are extracted; the connection information acquisition unit acquires the aperture as the connection information based on the thinned image and the extracted image; 3. The image processing device according to claim 2.

7. The link information acquisition unit extracting, from the thinned image, the thinned blood vessels that connect the two detection points that form the set; performing a dilation process on the extracted blood vessel and a filtering process using the extracted image as a filter to obtain the diameter; 7. The image processing device according to claim 6.

8. The linking information is Another detection point connected to the detection point; The number of the line segments connected to the detection point; and Including, 3. The image processing device according to claim 2.

9. the storage unit stores a plurality of filters corresponding to the crossover points, the branch points, and the end points, the detection unit detects the detection points by performing a filtering process on the thinned image using the filter.

2. The image processing device according to claim 1.

10. The thinned image is A specific pixel indicating the thinned blood vessel; non-specific pixels indicating a background other than the thinned blood vessels; It is composed of The group information identification unit In a first region of 3×3 pixels having the detection point as a first center pixel, the specific pixel is detected from eight pixels surrounding the first center pixel, and then: a process of detecting a new specific pixel from five pixels, excluding the first central pixel and a pixel adjacent to the first central pixel, among eight pixels surrounding the second central pixel in a second region of 3×3 pixels having the detected specific pixel as a second central pixel; a process of detecting a new specific pixel from five pixels, excluding the second center pixel and a pixel adjacent to the second center pixel, among eight pixels surrounding the third center pixel in a third region of 3×3 pixels having the detected specific pixel as a third center pixel; By repeating the above, the group information is identified.

2. The image processing device according to claim 1.

11. A computer is caused to function as the image processing device according to claim 1.

2. An image processing program comprising:

12. 1. An image processing method executed by an image processing device for identifying crossing points of blood vessels based on an image of the blood vessels, the method comprising: The image processing device includes: a storage unit that stores a thinned image in which the blood vessels included in the captured image are thinned; Equipped with The image processing device, detecting a plurality of detection points corresponding to any of the crossing points, branching points, and end points of the thinned blood vessels based on the thinned image; specifying, based on the thinned image, group information indicating a group of two of the detection points that are connected by the blood vessel, among the plurality of detection points detected; generating a blood vessel graph by connecting each of the two detection points constituting the pair with a line segment based on the identified pair information; acquiring connectivity information for each of the detected detection points based on the blood vessel graph; extracting a plurality of branch detection points connected to three of the line segments from among the plurality of detected detection points based on the connection information; extracting, as a detection point pair, two of the branch detection points that are connected to a common line segment from among the extracted branch detection points based on the connection information; identifying the branch detection points constituting the detection point pair as the detection points corresponding to the crossing point based on the connection information of the two branch detection points constituting the detection point pair; Including, 13. An image processing method comprising:

Images