Fundus image processing program and fundus photographing device

Abstract

TECHNICAL PROBLEM: To obtain a fundus image in which a distortion due to a fundus shape is excellently suppressed.SOLUTION: By executing a fundus image processing program by a processor of a fundus imaging apparatus 1, an acquisition step (S1), a correction step (S2) and a composition step (S3) are executed. In the acquisition step (S1), two or more fundus images which are captured via imaging optical systems 10, 20 at mutually-different imaging positions and partially overlap each other are acquired. In the correction step (S2), a portion or all of the image distortions included in the two or more fundus images are corrected on the basis of image information of the two or more fundus images which partially overlap each other. In the composition step (S3), the two or more fundus images whose distortions are corrected are combined to generate a panoramic fundus image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a fundus image processing program for processing a fundus image of a subject's eye, and a fundus photographing apparatus. [Background technology]

[0002] Non-mydriatic fundus cameras have been widely used as fundus photographing devices. Most non-mydriatic fundus cameras photograph fundus images with an angle of view of about 45°. As disclosed in Patent Document 1, a method is also known in which a wide range of the fundus is represented in a single panoramic image by panoramic synthesis of multiple fundus images with different fixation positions.

[0003] Meanwhile, in recent years, devices capable of photographing fundus images at wider angles have been attracting attention. For example, Patent Document 1 discloses a fundus photographing device that photographs color fundus images at a viewing angle of 90 degrees or more. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-123456 A Summary of the Invention [Problem to be solved by the invention]

[0005] Fundus images captured by fundus photography devices usually contain distortion. This distortion becomes greater toward the periphery of the image. Also, the wider the angle of view, the more noticeable the distortion becomes on the periphery.

[0006] Here, the distortion of the fundus image is caused by both the optical system of the device and the shape of the fundus. Of these, a correction formula for the distortion caused by the optical system of the device can be obtained in advance by photographing a chart or the like with a known shape. However, since the optical system of the eye varies from person to person, at least the distortion caused by the fundus shape cannot be corrected by the same method as the distortion caused by the optical system of the device.

[0007] Furthermore, when synthesizing a panoramic fundus image with an angle of view of about 45°, distortion caused by the fundus shape of the subject eye has not been considered a particular problem.

[0008] The present disclosure has been made in response to the problems of the conventional technology, and aims to provide a fundus image processing program and a fundus photographing device that can obtain fundus images in which distortion caused by the fundus shape is effectively suppressed. [Means for solving the problem]

[0009] A fundus image processing program according to a first aspect of the present disclosure is a fundus image processing program executed by a computer that processes data of a fundus image obtained by photographing the fundus of a subject's eye, and the fundus image processing program is executed by a processor of the computer, whereby the computer executes an acquisition step of acquiring two or more fundus images photographed via an imaging optical system at different imaging positions, the two or more fundus images being partially overlapped with each other, a correction step of correcting distortion of some or all of the images contained in the two or more fundus images based on image information of the two or more fundus images being partially overlapped with each other, and a synthesis step of generating a panoramic fundus image by synthesizing the two or more fundus images whose distortions have been corrected. Furthermore, the correction step includes a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other, and a coordinate transformation step of obtaining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points and applying the coordinate transformation to the fundus images in which the corresponding points have been set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

[0010] A fundus image processing program according to a second aspect of the present disclosure is a fundus image processing program executed by a computer that processes data of a fundus image obtained by photographing the fundus of a subject's eye, and the fundus image processing program is executed by a processor of the computer, whereby an acquisition step of acquiring two or more fundus images photographed via an imaging optical system at different photographing positions, the two or more fundus images being partially overlapping each other, and a correction step of correcting distortion of some or all of the images contained in the two or more fundus images based on image information of the two or more fundus images being partially overlapping each other are executed by the computer. Furthermore, the correction step includes a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other, and a coordinate transformation step of obtaining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points and applying the coordinate transformation to the fundus images in which the corresponding points have been set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

[0011] A fundus imaging apparatus according to a third aspect of the present disclosure includes the imaging optical system and an image processing apparatus that executes the fundus image processing program according to the first or second aspect. Effect of the Invention

[0012] According to the present disclosure, it is possible to obtain a fundus image in which distortion caused by the fundus shape is effectively suppressed. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram showing a schematic configuration of a fundus photographing device 1. FIG. [Diagram 2] FIG. 2 is a diagram showing a schematic configuration of a photographing optical system 2. [Diagram 3] FIG. 4 is a diagram showing the processing contents of each step of fundus image processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] "overview" First, a first embodiment that is an exemplary embodiment of the present disclosure will be described.

[0015] In the first embodiment, a method for generating a panoramic fundus image from a plurality of fundus images will be described. In the first embodiment, data of a fundus image of a subject's eye is processed by a computer. A fundus image processing program according to the first embodiment is executed by a processor of the computer, thereby executing an acquisition step, a correction step, and a synthesis step.

[0016] <Acquisition steps> In the acquisition step, two or more fundus images are acquired through an imaging optical system. The two or more acquired fundus images partially overlap each other. The two or more fundus images may be acquired at different fixation positions, resulting in two or more fundus images partially overlapping each other. Each fundus image is stored in a memory accessible by the processor.

[0017] The fundus image acquired in the acquisition step may be a front image (two-dimensional image) of the fundus. However, this is not necessarily limited to this, and the fundus image may be a three-dimensional image (volume data) in which depth information is added to the front image (two-dimensional image).

[0018] Each fundus image may be captured at an angle of view larger than 45°, which is the angle of view of a standard fundus camera.

[0019] <Correction step> In the correction step, based on the image information of two or more fundus images that overlap each other, the distortion of some or all of the images included in the two or more fundus images is corrected. That is, according to this embodiment, the image distortion is appropriately corrected without necessarily requiring any information on the optical system of the device or information on the fundus shape other than the fundus image. That is, not only the field curvature caused by the photographing optical system but also the influence of the distortion that differs for each subject eye caused by the curved shape of the fundus is suppressed.

[0020] The correction step may further include a corresponding point setting step and a coordinate conversion step as follows.

[0021] <Corresponding point setting step> In the corresponding point setting step, a plurality of corresponding points are set between two or more fundus images. In other words, a plurality of corresponding points are set in overlapping portions between two or more fundus images. The corresponding points may be set at positions of features common to the two or more fundus images. In the fundus images, the corresponding points may be set for structural features of tissues, such as branching of blood vessels.

[0022] Here, a method for setting a plurality of corresponding points between two or more fundus images can be appropriately selected. For example, the processor detects a plurality of points (feature points) on tissues for obtaining corresponding points from within an image area of ​​one of the two or more fundus images (referred to as a "target image"). For the detection of feature points, for example, points suitable for alignment may be detected using the Good Features To Track function of OpenCV or the like, or the feature points may be set at equal intervals. Next, from within an image area of ​​an image other than the target image (referred to as a "source image") among the two or more fundus images, corresponding points that correspond to the feature points of the target image are detected. For example, a template matching method, a block matching method, or the like can be used for the detection of corresponding points of the source image.

[0023] <Coordinate conversion step> Next, the coordinate transformation step will be described. For convenience, the fundus image to be processed in the coordinate transformation step may be referred to as an input image, and the processing result may be referred to as an output image.

[0024] In the coordinate transformation step, the processor determines a coordinate transformation that matches the positional relationship of the corresponding points between the fundus images, each of which has a corresponding point set thereon. For example, in the first embodiment, the processor determines a coordinate transformation (coordinate transformation formula) that minimizes the distance between the corresponding points in two or more input images under a constraint that takes into account the curvature of the field of the photographing optical system and the curvature of the fundus. The coordinate transformation (coordinate transformation formula) may be expressed by adding together a distortion correction model and a parallel translation (see the embodiments for a specific example).

[0025] Next, the obtained coordinate transformation is applied to two or more input images. In other words, based on the coordinate transformation formula obtained under the constraint conditions, the transformed coordinates for each pixel of the input image are obtained. As a result, an output image is obtained by providing the pixel value of each pixel of the input image to the transformed coordinates. At this time, the application of the coordinate transformation causes each input image to be deformed and moved. As a result, distortion is corrected in each fundus image, and further, alignment is performed.

[0026] In this embodiment, the constraint condition may be determined in advance regardless of the actual degree of the curvature of the field and the curvature of the fundus. Here, the image center of each fundus image is considered to have less distortion caused by the curvature of the fundus than the image peripheral portion. Also, the image center has less distortion caused by the field curvature of the photographing optical system than the image peripheral portion. Therefore, for each fundus image to be corrected for distortion, the constraint condition may be that "the coordinate change amount at the image center is suppressed more than the coordinate change amount at the image peripheral portion." More preferably, the constraint condition may be that "the coordinates at the image center are not changed by the distortion correction model." The image center may be an area at the image center of each fundus image, the position, shape, and size of which are determined in advance.

[0027] In addition, in a fundus image captured by tilting the visual axis with respect to the optical axis of the photographing optical system, it is considered that the region where the distortion caused by the curvature of the fundus is the least is displaced from the center of the image according to the amount of tilt between the optical axis and the visual axis. Therefore, within a range of the angle of view and tilt angle where the distortion in the center of the image is sufficiently small compared to the distortion in the peripheral part of the image, a constraint condition may be used that "the center of the image does not change in coordinates due to the distortion correction model." Alternatively, the image region where "there is no change in coordinates due to the distortion correction model" is specified as a constraint condition may be changed for each image according to the angle of view and tilt angle.

[0028] <Synthesis Steps> In the synthesis step, a panoramic fundus image is generated by synthesizing (combining) two or more fundus images in which the distortion has been corrected. If a coordinate transformation step has been performed in advance, two or more fundus images to which the coordinate transformation has been applied are synthesized. For overlapping parts, the pixel value of each pixel may be set by averaging or weighted averaging. In the case of weighted averaging, the closer to the edge of the fundus region in each fundus image, the smaller the weighting coefficient is set, so that the seams of the images are less noticeable. As a result of synthesis, a panoramic fundus image with a wider angle than the fundus image on which the synthesis is based can be obtained. Although the fundus image on which the synthesis is based had a larger distortion in the peripheral part of the image, the distortion is appropriately corrected by the correction step, so that discontinuous connection of the fundus structure at the seams between the images in the panoramic fundus image is appropriately reduced. In other words, by applying a coordinate transformation that makes the positional relationship of the multiple corresponding points coincide with each other between the multiple fundus images in which multiple corresponding points are set, discontinuity at the seams between the images is suppressed. In addition, distortion is corrected by coordinate transformation that minimizes the distance between multiple corresponding points in two or more input images under constraints that take into account the field curvature of the imaging optical system and the curvature of the fundus, thereby obtaining a panoramic fundus image that makes it easy to observe peripheral tissues.

[0029] <Second embodiment> The coordinate transformation in the correction step of the first embodiment is premised on obtaining a panoramic fundus image. If only the distortion of a part or all of the images included in the fundus image is corrected, it is not necessarily required to obtain a coordinate transformation (coordinate transformation formula) that "minimizes" the distance between multiple corresponding points in two or more input images. If only the distortion of a part or all of the images included in the fundus image is corrected, it is considered sufficient to obtain a coordinate transformation (coordinate transformation formula) that minimizes the variation in the distance between multiple corresponding points in two or more input images, and apply the obtained coordinate transformation to a part or all of the two or more fundus images in which corresponding points are set.

[0030] Even in the case where a panoramic fundus image is not necessarily generated and distortion correction is performed in this way, the descriptions given in <> in the first embodiment can be applied in part or in whole as appropriate. <Example> An example will be described below. In the embodiment, the fundus photographing device 1 itself photographs the fundus image and processes the photographed fundus image. That is, the fundus photographing device 1 of this embodiment functions as a computer that processes the fundus image. Therefore, the fundus photographing device 1 can process the photographed fundus image while photographing the fundus image. However, the device that can function as a computer is not limited to the fundus photographing device 1. For example, a personal computer, a server, a mobile terminal, or a smartphone that can acquire data of the fundus image photographed by the fundus photographing device 1 may function as a computer that performs the fundus image processing described below. In addition, the control units of multiple devices may cooperate to perform the fundus image processing.

[0031] A schematic configuration of a fundus photographing device 1 will be described with reference to FIG. 1 and FIG. 2. As shown in FIG. 1, the fundus photographing device 1 includes an optical unit 1A and a control unit 1B. The optical unit 1A stores a photographing optical system 2 (see FIG. 2). The control unit 1B includes a processor (CPU) 70 and a storage device (memory) 75. The processor 70 is a control unit that controls the fundus photographing device 1. A fundus image processing program is stored in the storage device 75. An operation unit 85 and a monitor 90 are connected to the control unit 1B. The operation unit 85 may be a pointing device such as a mouse or a touch panel, or may be another user interface. The monitor 90 can display various images such as a photographed fundus image. A personal computer (PC) or the like may be used as the control unit 1B. The optical unit 1A and the control unit 1B may be housed in the same housing or in separate housings.

[0032] The photographing optical system 2 will be described with reference to FIG. 2. The photographing optical system 2 illustrated in FIG. 2 is a scanning optical system. In detail, the photographing optical system 2 in this embodiment is a two-dimensional scanning type optical system. In the two-dimensional scanning type optical system, the illumination light is formed in a spot shape on the fundus and is scanned two-dimensionally on the fundus. However, a line scanning type (or slit scanning type) optical system may be adopted as the scanning type optical system. In this case, the illumination light is formed in a line shape or a slit shape on the fundus and is scanned in a direction intersecting with the cross-sectional longitudinal direction of the light beam of the illumination light. In addition, the photographing optical system does not have to be a scanning type optical system. For example, the photographing optical system may capture a two-dimensional front image of the fundus by simultaneously irradiating the illumination light to a two-dimensional area on the fundus. The photographing optical system 2 includes an illumination optical system 10 and a light receiving optical system 20.

[0033] The following describes the irradiation optical system 10. The irradiation optical system 10 includes a light source 11, a beam splitter 13, a lens 14, a scanning unit 16, and an objective lens 17.

[0034] In the irradiation optical system 10 of this embodiment, the illumination light focused in a point shape on the fundus is scanned by the scanning unit 16. The scanning unit 16 scans the illumination light in two different directions. For convenience of explanation, the following will exemplify a case in which the illumination light is scanned by raster scanning. In this case, the scanning unit 16 includes a first optical scanner 16A for main scanning and a second optical scanner 16B for sub-scanning. However, it is also possible to change the configuration of the scanning unit 16. For example, the scanning unit may be a two-degree-of-freedom device (e.g., MEMS, etc.). The optical scanner may be at least one of a galvanometer mirror, a polygon mirror, a resonant scanner, a MEMS, and an acousto-optical element (AOM), etc., as appropriate.

[0035] The irradiation optical system 10 irradiates the illumination light emitted by the light source 11 onto the fundus Er of the subject's eye E. The irradiation optical system 10 may be capable of simultaneously irradiating light in multiple wavelength ranges (multiple illumination lights having different wavelengths), or may be capable of selectively switching between multiple wavelength ranges to irradiate light. The irradiation optical system 10 of this embodiment is capable of switching between visible light and invisible light (infrared light in this embodiment) to irradiate the fundus Er. It is also capable of simultaneously irradiating the fundus Er with visible light in multiple wavelength ranges.

[0036] The beam splitter 13 combines and separates the optical path of the irradiation optical system 10 and the optical path of the light receiving optical system 20. The beam splitter 13 may be a perforated mirror or another member. The lens 14 is a focusing lens, and is displaced by a drive unit (not shown). The focus (visual acuity) is adjusted by displacing the lens 14 along the optical axis.

[0037] The objective lens 17 emits illumination light (laser light in this embodiment) scanned by the scanning unit 16 toward the fundus Er of the subject's eye E. The objective lens 17 forms a rotation point P at the position of the exit pupil. At the rotation point P, the illumination light that has passed through the scanning unit 16 is rotated. As a result, the illumination light is scanned over the fundus Er. When multiple illumination lights with different wavelengths are emitted simultaneously, the multiple illumination lights are irradiated onto the fundus Er along the same optical path that passes through the pupil of the subject's eye E. Fundus reflected light of the illumination light is emitted from the pupil as parallel light.

[0038] The light receiving optical system 20 will be described. In this embodiment, the light receiving optical system 20 shares with the irradiation optical system 10 each component arranged on the optical path from the objective lens 17 to the beam splitter 13. Light from the fundus Er (for example, fundus reflection light of illumination light, or fluorescence, etc.) is guided back along the optical path of the irradiation optical system 10 to the beam splitter 13. The beam splitter 13 guides the light from the fundus Er to an independent optical path of the light receiving optical system 20.

[0039] The independent optical path of the light receiving optical system 20 includes a lens 21, a pinhole plate 23, and a light separating unit 30. The pinhole plate 23 is disposed on the fundus conjugate plane, and removes light from positions other than the light collecting point or focal plane of the fundus Er, and guides light from the light collecting point or focal plane to the light separating unit 30. The light separating unit 30 separates light from the fundus Er. In this embodiment, the light separating unit 30 selectively separates light from the fundus Er in a wavelength-selective manner. That is, the light separating unit 30 branches the optical path of the light receiving optical system 20 according to the wavelength of light. As shown in FIG. 2, the light separating unit 30 of this embodiment may include a plurality of dichroic mirrors including two dichroic mirrors 31 and 32 having different optical separation characteristics (wavelength separation characteristics). The optical path of the light receiving optical system 20 is branched into three or more optical paths by the plurality of dichroic mirrors. One of the light receiving elements 25, 27, and 29 is disposed on each of the branched optical paths.

[0040] The light separation unit 30 of this embodiment separates the light from the fundus Er into wavelengths, and causes each of the three light receiving elements 25, 27, and 29 to receive light in a different wavelength range. In detail, each of the three light receiving elements 25, 27, and 29 receives one of three colors of light: blue, green, and red. Therefore, the fundus photographing device 1 can simultaneously photograph a plurality of (three in this embodiment) front fundus images photographed with illumination light of each of blue, green, and red wavelengths for the fundus Er. It is also possible to generate a color front fundus image from the plurality of front fundus images photographed simultaneously. The front fundus image is a two-dimensional image of the fundus as viewed from the optical axis direction of the illumination light.

[0041] As described above, the irradiation optical system 10 of this embodiment can switch between visible light and infrared light to irradiate the fundus Er. The fundus photographing device 1 can photograph an infrared image (two-dimensional front image) of the fundus Er by irradiating the fundus Er with infrared light and receiving the infrared light reflected from the fundus Er with a light receiving element. Since the subject does not feel glare while the infrared image is being photographed, there is little possibility that the quality of the infrared image will be deteriorated due to blinking or the like.

[0042] The fundus photographing device 1 further includes a fixation presenting optical system. In this embodiment, the irradiation optical system 10 also serves as the fixation presenting optical system. The fixation optical system presents a fixation target to the subject's eye. More specifically, visible light irradiated by the light irradiating unit 11 is used as the fixation target. In this case, the light emitting unit 10 may turn on the visible light at a predetermined scanning position and turn it off at other scanning positions to induce fixation of the subject's eye. The position of the fixation target can be changed by controlling the positions of the optical scanners 16A and 16B.

[0043] With reference to FIG. 3, the fundus image processing performed by the fundus photographing device 1 will be described. As described above, the fundus photographing device 1 is an example of a computer. Therefore, the device that performs the fundus image processing illustrated in FIG. 3 is not limited to the fundus photographing device 1. When the fundus image processing described below is performed by a device other than the fundus photographing device 1, the process of "photographing and acquiring" an image is replaced with a process of acquiring data of an image photographed by the fundus photographing device 1. In the fundus image processing of this embodiment, a panoramic fundus image is generated by combining multiple fundus images photographed by the photographing optical system 2.

[0044] For the sake of simplicity, a case will be described in which the number of fundus images used to construct a panoramic fundus image is the minimum of two. However, this is not necessarily limited to this, and a panoramic fundus image may be generated based on three or more fundus images.

[0045] 3, when fundus image processing is started, the processor 70 of the fundus photographing device 1 photographs fundus images at multiple photographing positions on the fundus of the same subject eye as a photographing target, and acquires image data (S1). This results in the acquisition of multiple fundus images photographed at different photographing positions.

[0046] For example, the photographing position is changed by changing the position of the fixation target presented to the eye to be examined. Here, it is assumed that two fundus images (5A, 5B) are acquired by photographing at two photographing positions different in the left-right direction. At this time, the photographing positions of the two fundus images 5A, 5B are set so that the images partially overlap each other. In addition, in this embodiment, as an example, the fundus image 5A is an image photographed in a state where the optical axis of the photographing optical system and the visual axis are aligned (that is, an image in which the fovea is located approximately at the center of the image), and the fundus image 5B is an image photographed at a position displaced in the left-right direction with respect to the fundus image 5A. However, this is not necessarily limited to this. In the following description, for the sake of simplicity, unless otherwise specified, it is assumed that the deviation of the photographing positions in the vertical direction is not taken into consideration (does not occur) when photographing the two photographed images 5A, 5B.

[0047] Next, the processor 70 executes a correction process (S2). In the correction process (S2), the distortion of the two fundus images 5A, 5B that partially overlap each other is corrected based on the image information of the fundus images. As a result of the correction process (S2) in this embodiment, not only are the distortions of the two fundus images 5A, 5B that partially overlap each other corrected, but also alignment is performed. However, this is not necessarily limited to this, and the correction of distortion and alignment may be performed step by step by separate processes.

[0048] The correction process (S2) will be described in detail below. In this embodiment, based on image information of two fundus images 5A and 5B captured at two different photographing positions in the left-right direction, the distortion components in the left-right direction, which are distortions of each image, are corrected by the correction process (S2).

[0049] First, the processor 70 sets a plurality of corresponding points 6a, 6b of two fundus images 5A, 5B. Here, the fundus image 5A is treated as a target image, and the fundus image 5B is treated as a source image. First, the processor 70 detects a plurality of points (feature points) on the tissue for obtaining the corresponding points from within the image area of ​​the fundus image 5A, which is the target image. As an example, the well-known OpenCV Good Features To Track function or the like is used to detect the feature points. As a result, a plurality of feature points are detected so as to be approximately uniform over the entire image area of ​​the fundus image 5A. In detail, in this embodiment, a plurality of feature points are arranged so that the number of feature points in a unit area is within a predetermined range.

[0050] Next, the processor 70 detects corresponding points 6b corresponding to the feature points of the fundus image 5A, which is the target image, from within the image area of ​​the frontal fundus image 5B, which is the source image. The detection of the corresponding points 6b can be performed using, for example, template matching or block matching. Among the multiple feature points set in the fundus image 5A, some of the feature points that have corresponding points 6b in the fundus image 5B are treated as corresponding points 6a in the fundus image 5A.

[0051] Next, a coordinate transformation equation that minimizes the distance between corresponding points on two fundus images 5A and 5B (i.e., the distance between corresponding points 6a and 6b set to the same feature) is obtained. In this embodiment, the coordinate transformation equation is expressed by adding together an equation representing a distortion correction model and an equation representing parallel translation. The equation representing the distortion correction model in this embodiment is a distortion correction equation. The coordinate transformation equation for point i is shown as [Equation 1].

[0052]

number

[0053] In the formula 1, "x ai (1+k a1 r ai 2 +k a2 r ai 4+ k a3 r ai 6 )” represents the distortion correction model. a represents the translation. Note that x ai is the x coordinate of point i in the fundus image 5A, and x ai ' is the x-coordinate of point i in the transformed fundus image 5A', and r ai is the distance from the origin of the point i in the fundus image 5A. The origin here is the center of the image. The coordinate transformation equation for the fundus image 5B can also be expressed in a similar format to [Equation 1].

[0054] In this embodiment, the coefficients of the above coordinate transformation equation are solved using the least squares method. Minimizing the distance between corresponding points can be expressed as [Equation 2]. By deriving the coefficients of [Equation 2], it is possible to obtain a coordinate transformation equation that minimizes the distance between corresponding points on two fundus images 5A and 5B.

[0055]

number

[0056] However, simply minimizing the distance between corresponding points may result in the two fundus images 5A and 5B being inappropriately deformed and aligned with respect to the actual distortion. Therefore, in this embodiment, the coefficients are derived by solving the constrained least squares method based on the following constraints: Constraint 1: The coordinates of any one image do not change when translated. · Constraint 2: The coordinates of the center of the image do not change due to the distortion correction model.

[0057] <Regarding constraint 1> Constraint condition 1 is derived from the need to determine an appropriate image as a reference when aligning multiple images. In this embodiment, by excluding translation from the coordinate transformation of the fundus image 5A, the origin of the fundus image 5A is fixed as the origin of the transformed image.

[0058] <Regarding constraint 2> Near the optical axis of the photographing optical system, it is considered that there is almost no curvature of field of the photographing optical system and almost no displacement of the fundus surface in the optical axis direction, regardless of individual differences in the shape of the subject eye and the angle between the optical axis of the photographing optical system and the visual axis, etc. When the photographing position is changed by changing the positional relationship between the optical axis of the photographing optical system and the visual axis as in this embodiment, the image center of each fundus image 5A, 5B is the area with the least distortion in each image, and thus constraint condition 2 is derived.

[0059] Constraints 1 and 2 can be expressed, for example, by the following equations. Based on constraint 1, [Equation 3] is derived. According to constraint 2, the coordinates do not move near the center (x=0.1) of each of fundus images 5A and 5B due to distortion correction, so [Equation 4] and [Equation 5] are derived, respectively.

[0060]

number

[0061]

number

[0062]

number

[0063] The coordinate transformation equations obtained by solving [Equation 2] to [Equation 5] are applied to each pixel of each fundus image 5A, 5B, and the coordinates of each pixel after transformation can be obtained. In this way, the pixel values ​​of each pixel in the input image (fundus images 5A, 5B before transformation) are assigned to the coordinates after transformation, respectively, to obtain an output image. Here, for pixels in the output image to which no pixel value has been assigned, the pixel value can be obtained by interpolating the surrounding pixels.

[0064] Image distortion occurs not only in the left-right direction (x direction) that is the direction connecting the photographing positions of the two fundus images 5A and 5B, but also in the up-down direction (y direction) that is perpendicular to the left-right direction. A conversion formula for the y direction can be obtained in a similar manner, but when only images with different photographing positions in the left-right direction are handled as in this embodiment, it is difficult to appropriately correct the distortion in the up-down direction. Therefore, for example, in one fundus image (preferably, the fundus image 5A photographed with the optical axis of the photographing optical system and the visual axis aligned; an image in which the fovea is located approximately at the center of the image), the distortion in the x direction and the distortion in the y direction may be assumed to be equivalent, and a coordinate conversion in the y direction for each y coordinate of the fundus images 5A and 5B may be obtained.

[0065] In this embodiment, in this manner, each pixel in the fundus images 5A and 5B is moved based on the coordinate transformation equation obtained under the constraint conditions.

[0066] In detail, the distortion in each of the fundus images 5A and 5B is corrected based on the equation of the distortion correction model. As described above, the image center is transformed to minimize the distance between the corresponding points of the fundus images 5A and 5B under the condition that the coordinates of the image center are not changed by the distortion correction model (condition 1), so that the distortion that increases the further away from the image center is appropriately suppressed. At this time, the image peripheral parts of the fundus images 5A and 5B after transformation are elongated compared to the image center parts of the fundus images 5A and 5B before transformation. In this way, in this embodiment, the distortion in the fundus images 5A and 5B can be appropriately corrected without necessarily requiring either the information on the optical system of the device or the information on the fundus shape. That is, not only the effect of the field curvature caused by the photographing optical systems 10 and 20, but also the effect of the distortion that differs for each subject eye caused by the curved shape of the fundus is suppressed. Furthermore, alignment is performed based on the equation of translation.

[0067] After the correction process is performed as described above, the fundus images 5A and 5B are combined (S3). This results in a panoramic fundus image based on the two fundus images 5A and 5B. The fundus images 5A and 5B used as the basis for the synthesis had larger distortions in the peripheral areas of the images, but the distortions are appropriately corrected by the correction step (S2), so that the unnaturalness of the seams between the images in the panoramic fundus image is appropriately reduced. In addition, the converted fundus images 5A and 5B used as the basis for the synthesis have larger distortions in the peripheral areas stretched compared to the images before the conversion, so that a panoramic fundus image in which the tissues in the peripheral areas are easily observed can be obtained.

[0068] <Modification> The techniques disclosed in the above embodiments are merely examples, and therefore the techniques exemplified in the above embodiments can be modified.

[0069] For example, in the above embodiment, the distortion correction is performed based on two fundus images 5A and 5B, and a panoramic fundus image is generated. However, this is not necessarily limited to this, and for example, the distortion correction may be performed based on three or more fundus images, and a panoramic fundus image may be generated.

[0070] For example, in addition to the two fundus images 5A and 5B shown in the above embodiment, at least a third fundus image 5C may be used. The third fundus image 5C is a fundus image captured at a different capturing position in the y direction from the two fundus images 5A and 5B, and may be an image that partially overlaps with at least one of the two fundus images 5A and 5B.

[0071] When three fundus images 5A, 5B, and 5C are present as input images, the coordinate transformation of the three images may be solved simultaneously. That is, in addition to minimizing the distance between corresponding points of fundus images 5A and 5B as shown in the above embodiment, the coordinate transformation coefficients of each image that satisfy the minimization of the distance between corresponding points of fundus images 5A and 5C are obtained. Here, in addition to the constraints shown in the above embodiment, the coordinate transformation coefficients of each image are obtained under the constraint that there is no distortion near the center of fundus image 5C. By performing this for each of the x and y directions, it is possible to obtain a coordinate transformation formula used for correcting the distortion of 5A, 5B, and 5C and for positioning.

[0072] In this method, correction of distortion in the y direction is performed based on coordinate transformation using image information from a third fundus image 5C, which is taken at a different shooting position in the y direction from the two fundus images 5A and 5B, making it easier to properly correct distortion in accordance with the actual shape of the test eye. [Explanation of symbols]

[0073] 1 Fundus photography device

Claims

1. A fundus image processing program executed by a computer that processes fundus image data obtained by photographing a fundus of a subject's eye, The fundus image processing program is executed by the processor of the computer, acquiring two or more fundus images captured at different photographing positions through an imaging optical system, the two or more fundus images being partially overlapped with each other; a correction step of correcting distortion of a part or all of images included in two or more fundus images that partially overlap each other based on image information of the two or more fundus images; a synthesis step of generating a panoramic fundus image by synthesizing two or more fundus images in which the distortion has been corrected, the synthesis step being executed by the computer; Furthermore, the correction step a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other; a coordinate transformation step of determining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points based on coordinate information of the plurality of corresponding points, and applying the coordinate transformation to the fundus image in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

2. A fundus image processing program executed by a computer that processes fundus image data obtained by photographing a fundus of a subject's eye, The fundus image processing program is executed by the processor of the computer, acquiring two or more fundus images captured at different photographing positions through an imaging optical system, the two or more fundus images being partially overlapped with each other; a correction step of correcting distortion of a part or all of images included in two or more fundus images that partially overlap each other based on image information of the two or more fundus images; is executed by the computer, Furthermore, the correction step a corresponding point setting step of setting a plurality of corresponding points between two or more fundus images that partially overlap each other; a coordinate transformation step of determining a coordinate transformation that minimizes the variation in distance between the plurality of corresponding points based on coordinate information of the plurality of corresponding points, and applying the coordinate transformation to the fundus image in which the corresponding points are set, thereby correcting distortion of some or all of the images contained in the two or more fundus images based on the set corresponding points.

3. A fundus image processing program as described in claim 1 or 2, wherein in the correction step, the coordinate transformation is performed under the constraint that, for each fundus image to be corrected for distortion, the amount of coordinate change in the center of the image is suppressed more than the amount of coordinate change in the peripheral part of the image.

4. The photographing optical system; A fundus photographing apparatus comprising: an image processing apparatus that executes the fundus image processing program according to claim 1 .

Images