Fundus image stitching method, apparatus, and electronic device

Abstract

The present application relates to the technical field of image stitching, and in particular to a fundus image stitching method, an apparatus, and an electronic device. The method comprises: acquiring at least two fundus images collected by an OCT device, the at least two fundus images corresponding to different fundus positions of an examined eye; determining from the at least two fundus images a reference image and images to be stitched other than the reference image; and respectively registering each image to be stitched with the reference image, and calculating a first overlapping area between each image to be stitched and the reference image. The fundus image stitching method provided by the present application improves the uniformity of feature point extraction by segmenting the overlapping area to obtain feature pixel points, and reduces the optical distortion by means of regional registration, thereby obtaining a target stitched image having good continuity and high stitching accuracy.

Description

Fundus image stitching method, device and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202311767495.6, filed with the Chinese Patent Office on December 20, 2023, entitled “A method, device and electronic device for fundus image stitching”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the technical field of image stitching, and in particular to a fundus image stitching method, device and electronic equipment. Background Art

[0004] Fundus image stitching technology is an image processing technology that combines fundus information, aiming to provide more comprehensive fundus information to help doctors more accurately assess eye conditions and related diseases. This technology forms a continuous retinal image by aligning, fusing, and edge-processing multiple fundus images to provide more comprehensive and continuous blood flow information. The main advantage of fundus image stitching technology is that it provides a panoramic retinal image that can display the blood flow situation of the entire retinal area. In summary, fundus image stitching technology provides ophthalmologists with more comprehensive and continuous retinal image information, which helps improve the accuracy of disease diagnosis and promotes more effective treatment decisions. It has a wide range of applications in fields such as diabetic retinopathy, retinal vascular disease, and glaucoma.

[0005] However, in the existing image stitching process, due to the optical distortion and uneven feature point extraction of images collected at different positions, the stitched image has poor continuity and accuracy. Summary of the Invention

[0006] In view of this, the purpose of the present application is to provide a fundus image stitching method, device and electronic equipment to improve the continuity and accuracy of fundus image stitching.

[0007] In a first aspect, an embodiment of the present application provides a fundus image stitching method, the method comprising:

[0008] Obtain at least two fundus images captured by an OCT device, wherein the at least two fundus images correspond to different fundus positions of the inspected eye;

[0009] Determining a reference image and an image to be stitched other than the reference image from at least two fundus images;

[0010] Registering each image to be stitched with the reference image, and calculating a first overlapping area between each image to be stitched and the reference image;

[0011] Segmenting the key area of ​​each image to be stitched to obtain at least two image blocks, and extracting feature pixel points corresponding to each image block; wherein the key area includes the first overlapping area;

[0012] For each of the multiple feature pixel points corresponding to the image to be stitched, calculate the target transformation matrix from each image to be stitched to the reference image;

[0013] Based on the target transformation matrix corresponding to each image to be stitched, multiple images to be stitched are stitched with the reference image to obtain a target stitched image.

[0014] In one embodiment, the reference image includes a central fundus image obtained by performing blood flow imaging on the central area of ​​the fundus, or includes an image of both the macula and the optic disc.

[0015] In one embodiment, the at least two fundus images include an image corresponding to a central position of the fundus and an image corresponding to a peripheral position of the fundus; and the step of determining a reference image and an image to be stitched other than the reference image from the at least two fundus images includes:

[0016] The reference image is obtained by subtracting the low-pass filtering result from the image corresponding to the fundus center position, and the image corresponding to the fundus circumferential position is subtracted from the low-pass filtering result to obtain the image to be stitched corresponding to each fundus circumferential position image.

[0017] In one embodiment, the step of registering each image to be stitched with a reference image and calculating a first overlapping area between each image to be stitched and the reference image includes:

[0018] Registering each image to be stitched with the reference image, and calculating a first offset between the image to be stitched and the reference image;

[0019] Based on the first offset, a first overlapping area between each image to be stitched and the reference image is calculated.

[0020] In one embodiment, the step of extracting characteristic pixel points corresponding to each image block includes:

[0021] The feature pixel points corresponding to each image block are extracted based on feature extraction rules; wherein the feature extraction rules include one or more of the following: surrounding gradient rules, grayscale rules, brightness rules, and texture rules of the feature pixel points.

[0022] In one embodiment, the step of calculating a target transformation matrix from each image to be stitched to a reference image for a plurality of feature pixels corresponding to each image to be stitched includes:

[0023] Obtaining first coordinates of multiple characteristic pixel points corresponding to each image to be stitched in the image to be stitched and second coordinates of multiple characteristic pixel points in the reference image;

[0024] Calculate the target transformation matrix from each image to be stitched to the reference image based on the first coordinate and the second coordinate.

[0025] In one embodiment, the step of calculating a target transformation matrix from each image to be stitched to a reference image based on the first coordinate and the second coordinate includes:

[0026] Calculate the target transformation matrix according to the following formula:

[0027] Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel point on the nth image to be stitched, and (x, y) is the second coordinate of the feature pixel point on the reference image.

[0028] In one embodiment, the step of stitching each image to be stitched with a reference image based on a target transformation matrix corresponding to each image to be stitched to obtain a target stitched image includes:

[0029] For each image to be stitched, calculate the predicted pixel point corresponding to each feature pixel point in the image to be stitched and mapped to the target stitching image according to the corresponding target transformation matrix;

[0030] For each predicted pixel, determining a feature pixel associated with the predicted pixel in each image to be stitched;

[0031] The pixel values ​​of each feature pixel associated with the predicted pixel are weighted averaged to calculate the target pixel value and obtain the target spliced ​​image.

[0032] In one embodiment, the step of taking a weighted average of the pixel values ​​of each feature pixel associated with the predicted pixel to calculate the target pixel value and obtain the target stitched image includes:

[0033] The target pixel value of each predicted pixel is calculated according to the following formula to obtain the target spliced ​​image:

[0034] Among them, dst is the target pixel value of the predicted pixel in the target spliced ​​image, src n is the pixel value of the feature pixel associated with the predicted pixel on the nth image to be stitched, w n src n The corresponding impact factor.

[0035] In one embodiment, before the step of stitching each image to be stitched with a reference image based on a target transformation matrix corresponding to each image to be stitched to obtain a target stitched image, the step further includes:

[0036] For each image to be stitched, calculate the overlapping area between each image to be stitched and the reference image according to the target transformation matrix;

[0037] Perform histogram matching on the image to be stitched and the reference image in the overlapping area, and adjust the image to be stitched according to the result of the histogram matching.

[0038] In one embodiment, the step of adjusting the images to be stitched according to the result of histogram matching includes:

[0039] The brightness and / or contrast of the image to be stitched is adjusted according to the result of the histogram matching, so that the image to be stitched is consistent with the reference image in brightness and / or contrast.

[0040] In one embodiment, there are at least two images to be stitched;

[0041] After the steps of registering each image to be stitched with the reference image and calculating the first overlapping area between each image to be stitched and the reference image, the method further includes:

[0042] A second overlapping area between the images to be stitched is calculated according to the first offset corresponding to each image to be stitched.

[0043] In one embodiment, the step of calculating the second overlapping area between the images to be stitched according to the first offset corresponding to each image to be stitched includes:

[0044] Determining a first offset between the first image to be stitched and the reference image, and a second offset between the second image to be stitched and the reference image;

[0045] A second overlapping area between the first image to be stitched and the second image to be stitched is calculated based on the second offset and the first offset.

[0046] In one embodiment, the step of segmenting the key area of ​​each image to be stitched to obtain at least two image blocks includes:

[0047] The first overlapping area and the second overlapping area corresponding to each image to be stitched are divided to obtain a plurality of image blocks.

[0048] In one embodiment, the step of calculating the target transformation matrix from each image to be stitched to the reference image for all feature pixels corresponding to each image to be stitched includes:

[0049] According to the second overlapping area between the images to be stitched, obtaining third coordinates of a plurality of characteristic pixel points corresponding to each image to be stitched on the second overlapping area;

[0050] Calculate the target transformation matrix from each image to be stitched to the reference image based on the first coordinate, the second coordinate, and the third coordinate.

[0051] In one embodiment, the step of calculating a target transformation matrix from each image to be stitched to a reference image based on the first coordinate, the second coordinate, and the third coordinate includes:

[0052] The target transformation matrix of each image to be stitched and the reference image is calculated using the following formula:

[0053] Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, T m is the target transformation matrix from the mth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel on the nth image to be stitched, (x, y) is the second coordinate of the feature pixel on the reference image, (x m ,y m ) is the third coordinate of the feature pixel point on the mth image to be stitched.

[0054] In a second aspect, the present application provides a fundus image processing device, comprising:

[0055] an acquisition module configured to acquire at least two fundus images acquired by an OCT device, wherein the at least two fundus images correspond to different fundus positions of the eye to be inspected;

[0056] a determination module configured to determine a reference image and an image to be stitched other than the reference image in at least two fundus images;

[0057] a registration module configured to register each image to be stitched with the reference image and calculate a first overlapping area between each image to be stitched and the reference image;

[0058] a feature pixel point extraction module configured to segment a key area of ​​each image to be stitched to obtain at least two image blocks, and extract feature pixel points corresponding to each image block; wherein the key area includes the first overlapping area;

[0059] a transformation matrix calculation module configured to calculate a target transformation matrix from each image to be stitched to a reference image for a plurality of feature pixels corresponding to each image to be stitched;

[0060] The image stitching module is configured to stitch each image to be stitched with a reference image based on a target transformation matrix corresponding to each image to be stitched, to obtain a target stitched image.

[0061] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the above method.

[0062] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. A processor executes the computer program to implement the method as described above.

[0063] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0064] The embodiments of the present application bring the following beneficial effects: The present application provides a fundus image stitching method, device and electronic device, the method comprising: obtaining at least two fundus images captured by an OCT device, the at least two fundus images corresponding to different fundus positions of the eye to be inspected; determining a reference image and an image to be stitched other than the reference image in the at least two fundus images; registering each image to be stitched with the reference image respectively, and calculating a first overlapping area between each image to be stitched and the reference image; segmenting a key area of ​​each image to be stitched to obtain at least two image blocks, and extracting feature pixel points corresponding to each image block; wherein the key area includes the first overlapping area; calculating a target transformation matrix from each image to be stitched to the reference image for multiple feature pixel points corresponding to each image to be stitched; stitching each image to be stitched with the reference image based on the target transformation matrix corresponding to each image to be stitched to obtain a target stitched image.

[0065] The fundus image stitching method provided in this application pre-registers each image to be stitched with a reference image, calculates the overlapping area between each pair of images, segments the overlapping area, extracts feature pixels, calculates the target transformation matrix between each image to be stitched and the reference image based on the multiple feature pixels, and then stitches each image to be stitched with the reference image based on the target transformation matrix to obtain the target stitched image. By segmenting the overlapping area and obtaining feature pixels, the uniformity of feature point extraction is improved, and optical distortion is reduced through regional registration, resulting in a target stitched image with good continuity and high stitching accuracy.

[0066] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description or be understood by practicing the present application. The objectives and other advantages of the present application are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0067] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0069] FIG1 is a flow chart of a fundus blood flow effect splicing method provided in an embodiment of the present application;

[0070] FIG2 is a schematic diagram of an original image collected in the fundus blood flow effect stitching method provided in an embodiment of the present application;

[0071] FIG3 is a schematic diagram obtained after low-pass filtering of FIG2 ;

[0072] FIG4 is a schematic diagram of a reference image in a fundus blood flow effect stitching method provided in an embodiment of the present application;

[0073] FIG5 is a schematic diagram of an image to be stitched obtained after preprocessing an initial image at the upper left position of the fundus center provided in an embodiment of the present application;

[0074] FIG6 is a schematic diagram of an image to be stitched obtained after preprocessing an initial image at the lower left position of the fundus center provided in an embodiment of the present application;

[0075] FIG7 is a schematic diagram of an image to be stitched obtained after preprocessing an initial image at the upper right position of the fundus center provided in an embodiment of the present application;

[0076] FIG8 is a schematic diagram of an image to be stitched obtained after preprocessing an initial image at the lower right position of the fundus center provided in an embodiment of the present application;

[0077] FIG9 is a schematic diagram of a target stitched image obtained by stitching FIG4 , FIG5 , FIG6 , FIG7 , and FIG8 according to the fundus image stitching method provided in an embodiment of the present application;

[0078] FIG10 is a schematic diagram of the structure of the fundus image stitching device provided in an embodiment of the present application.

[0079] Reference numerals: 10 - acquisition module, 20 - determination module, 30 - registration module, 40 - feature pixel extraction module, 50 - transformation matrix calculation module, 60 - image stitching module. DETAILED DESCRIPTION

[0080] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0081] To facilitate understanding of this embodiment, the following is a brief introduction to the technical terms used in this application.

[0082] The fundus image may be a vascular image obtained by OCTA or a structural image obtained by OCT scanning. The structural image may be a structural projection image or a B-scan image.

[0083] After introducing the technical terms involved in this application, the application scenarios and design concepts of the embodiments of this application are briefly introduced.

[0084] Fundus image stitching technology is widely used in ophthalmology. In diabetic retinopathy, by integrating multiple fundus images, doctors can more accurately assess characteristics such as vascular abnormalities, ischemic areas, and angiogenesis, helping to determine treatment plans. In retinal vascular diseases, blood flow stitching technology can provide information on blood flow parameters such as blood flow velocity, vascular density, and blood flow direction, which helps to assess the severity and progression of vascular lesions. In glaucoma treatment, fundus image stitching technology can help evaluate the effects of intraocular pressure regulation and blood flow improvement, guiding treatment decisions. It has a wide range of applications in fields such as diabetic retinopathy, retinal vascular diseases, and glaucoma.

[0085] In the image stitching process of related technologies, due to the optical distortion and uneven feature extraction of images collected at different positions, the continuity of interpolation and stitching accuracy of the final stitched image need to be improved.

[0086] Based on this, the present application provides a fundus image stitching method, device and electronic device. The method is applied to an electronic device in an image stitching system. The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the fundus image stitching method provided in the embodiment of the present application to improve the continuity and accuracy of fundus image stitching.

[0087] Example 1

[0088] This embodiment provides a fundus image stitching method, as shown in FIG1 , comprising:

[0089] S110 , obtaining at least two fundus images captured by an OCT device, wherein the at least two fundus images correspond to different fundus positions of the eye to be inspected.

[0090] S120 : Determine a reference image and an image to be stitched other than the reference image in at least two fundus images.

[0091] S130 , registering each image to be stitched with the reference image, and calculating a first overlapping area between each image to be stitched and the reference image.

[0092] S140 , segmenting the key area of ​​each image to be stitched to obtain at least two image blocks, and extracting feature pixel points corresponding to each image block; wherein the key area includes the first overlapping area.

[0093] S150 , calculating a target transformation matrix from each image to be stitched to a reference image for a plurality of characteristic pixel points corresponding to each image to be stitched.

[0094] S160 , based on the target transformation matrix corresponding to each image to be stitched, stitch each image to be stitched with the reference image to obtain a target stitched image.

[0095] In this embodiment, the image to be stitched and the reference image are sequentially registered to obtain a first overlapping area. Then, the first overlapping area and other key areas are segmented and feature points are extracted to obtain evenly distributed feature points, thereby reducing inaccuracies in stitching results caused by optical distortion and different contrasts of the images to be stitched, thereby improving the accuracy and precision of image stitching.

[0096] Specifically, the at least two fundus images acquired in step S110 may include a central fundus image acquired at the center of the fundus and an image corresponding to a peripheral fundus position. The image corresponding to the peripheral fundus position may be an image obtained by performing blood flow imaging on the fundus edge or an area adjacent to the central fundus region. The fundus image may be a vascular image acquired by OCTA or a structural image acquired by OCT scanning. The structural image may be a structural projection image or a B-scan image.

[0097] In one possible implementation, the at least two fundus images may include an image corresponding to a central position of the fundus and an image corresponding to a peripheral position of the fundus. Based on this, step S120 of determining a reference image and an image to be stitched excluding the reference image from the at least two fundus images may specifically include:

[0098] The reference image is obtained by subtracting the low-pass filtering result from the image corresponding to the fundus center position, and the image corresponding to the fundus circumferential position is subtracted from the low-pass filtering result to obtain the image to be stitched corresponding to each fundus circumferential position image.

[0099] The fundus center image obtained by performing blood flow imaging on the fundus center area, ie, the reference image, may also be an image including both the macula and the optic disc.

[0100] In a possible implementation, step S130 registers each image to be stitched with the reference image and calculates a first overlapping area between each image to be stitched and the reference image, which may specifically include:

[0101] S131 , registering each image to be stitched with a reference image, and calculating a first offset between the image to be stitched and the reference image.

[0102] S132: Calculate a first overlapping area between each image to be stitched and the reference image based on the first offset.

[0103] Specifically, since the image corresponding to the circumferential position of the fundus is collected at the circumferential position of the fundus center image with the fundus center image as the center, within a certain collection radius, the corresponding image to be stitched and the reference image have a local overlapping area. By selecting the position coordinates of a feature point in the image to be stitched and the feature point in the reference image, the first offset of the feature point in the image to be stitched and the reference image is calculated using the two position coordinates, and the first overlapping area is calculated and determined based on the first offset.

[0104] In one possible implementation, extracting the feature pixels corresponding to each image block in step S140 may include extracting the feature pixels corresponding to each image block based on a feature extraction rule, where the feature extraction rule may include one or more of the following: a surrounding gradient rule, a grayscale rule, a brightness rule, and a texture rule for the feature pixels. It should be noted that one or more feature pixels may be extracted from each image block.

[0105] In a possible implementation, step S150 calculates a target transformation matrix from each image to be stitched to a reference image for a plurality of feature pixels corresponding to each image to be stitched, which may specifically include:

[0106] S151 , obtaining first coordinates of a plurality of characteristic pixel points corresponding to each image to be stitched in the image to be stitched and second coordinates of the plurality of characteristic pixel points in the reference image.

[0107] S152 , calculating a target transformation matrix from each image to be stitched to the reference image according to the first coordinate and the second coordinate.

[0108] It is worth noting that the multiple feature pixels can be all the feature pixels, or multiple feature pixels with high confidence or pixel values ​​greater than a certain threshold among all the feature pixels, or they can be manually selected based on experience, which is not limited here.

[0109] After segmenting key areas such as the first overlapping area and extracting multiple feature pixels based on feature extraction rules, the target transformation matrix between each image to be stitched and the reference image can be calculated based on the different coordinates of the feature pixels on each image to be stitched and the reference image. Specifically, the target transformation matrix can be calculated using the following formula:

[0110] Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel point on the nth image to be stitched, and (x, y) is the second coordinate of the feature pixel point on the reference image.

[0111] In a possible implementation, step S160 stitches each image to be stitched with the reference image based on the target transformation matrix corresponding to each image to be stitched to obtain a target stitched image. Specifically, the following steps may be performed:

[0112] S161 , for each image to be stitched, calculating, according to the corresponding target transformation matrix, a predicted pixel point corresponding to each feature pixel point in the image to be stitched mapped to the target stitched image.

[0113] S162 : For each predicted pixel point, determine a feature pixel point associated with the predicted pixel point in each image to be spliced.

[0114] S163 , taking a weighted average of the pixel values ​​of each feature pixel associated with the predicted pixel, calculating a target pixel value, and obtaining a target spliced ​​image.

[0115] After calculating the target transformation matrix from each image to be stitched to the reference image in step S150, the feature pixels in the image to be stitched can be mapped to the reference image via Gaussian positive mapping. At this point, the target pixel value of the predicted pixel in the reference image corresponding to the feature pixel is calculated. Specifically, since the Gaussian positive mapping of a feature pixel (u, v) affects the predicted pixel and the points within a set range of the predicted pixel, that is, multiple predicted pixels within a range of the reference image corresponding to the feature pixel, in this embodiment, the set range is a 3×3 pixel matrix, and the influence factor ω conforms to the exp(-100×dis) distribution, where dis is the Euclidean distance from the corresponding range to the feature pixel (u, v). Similarly, the predicted pixel should be affected by multiple feature pixels in the corresponding n images to be stitched. Therefore, the feature pixels associated with the predicted pixel are determined and weighted averaged to calculate the target pixel value.

[0116] Specifically:

[0117] Among them, dst is the target pixel value of the predicted pixel in the target spliced ​​image, src n is the pixel value of the feature pixel associated with the predicted pixel on the nth image to be stitched, w n src n The corresponding impact factor.

[0118] In a possible implementation, before step S160 stitches each image to be stitched with the reference image based on the target transformation matrix corresponding to each image to be stitched to obtain the target stitched image, the following steps may be further included:

[0119] S170 , for each image to be stitched, calculating an overlapping area between each image to be stitched and the reference image according to the target transformation matrix.

[0120] S180 , performing histogram matching on the image to be stitched and the reference image in the overlapping area, and adjusting the image to be stitched according to the result of the histogram matching.

[0121] Based on the calculated target transformation matrix corresponding to each image to be stitched, the overlapping area between each image to be stitched and the reference image is accurately calculated to determine the size of the stitched image. The brightness and / or contrast of the image to be stitched are adjusted according to the results of histogram matching to make the brightness and / or contrast of the image to be stitched consistent with the reference image.

[0122] Example 2

[0123] Compared with Example 1, the present embodiment differs in that, in the present embodiment, step S110 obtains at least two fundus images captured by the OCT device, including at least two images corresponding to the circumferential positions of the fundus. Exemplarily, the initial images are four, corresponding to the upper left position, lower left position, upper right position and lower right position of the fundus center, respectively.

[0124] In this embodiment, after executing steps S131-S132, step S130 further includes:

[0125] S133 , calculating a second overlapping area between the images to be stitched according to the first offset corresponding to each image to be stitched.

[0126] Specifically, step S133 may include: determining a first offset between the first image to be stitched and the reference image, and a second offset between the second image to be stitched and the reference image; and calculating a second overlapping area between the first image to be stitched and the second image to be stitched based on the second offset and the first offset.

[0127] Similarly, if the first offset between the first image to be stitched and the reference image is known, and the second offset between the other images to be stitched and the reference image is also known, the second overlapping area between the first image to be stitched and the other images to be stitched can be calculated.

[0128] In this step, each image to be stitched and the reference image are pre-registered, and any two images to be stitched are pre-registered to estimate the first overlapping area and the second overlapping area.

[0129] In one possible implementation, the key area also includes a second overlapping area. The first overlapping area and the second overlapping area corresponding to each image to be stitched can be segmented to obtain multiple image blocks. The first overlapping area and the second overlapping area can be segmented to obtain a large number of subdivided small blocks. For each small block, multiple feature pixels are extracted based on feature extraction rules, so that the extracted feature points are relatively evenly distributed in the first overlapping area and the second overlapping area. Similarly, the feature extraction rules can include one or more of the following: a gradient rule surrounding the feature pixel, a grayscale rule, a brightness rule, and a texture rule.

[0130] In one possible implementation, the target transformation matrix between the first image to be stitched and the reference image is calculated based on the different coordinate values ​​of the feature pixel points on the first image to be stitched and the reference image. Specifically, the following registration formula is used for calculation:

[0131] Among them, T n is the first target transformation matrix from the nth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel point on the nth image to be stitched, (x, y) is the second coordinate of the feature pixel point on the reference image, and the first target transformation matrix is ​​determined as the target transformation matrix corresponding to each image to be stitched.

[0132] In another possible implementation, combined with the above example, since the images corresponding to the fundus circumferential positions are acquired at circumferential positions centered on the fundus center image, each corresponding image to be stitched has a partial overlap with the reference image, but a large part of the area does not overlap. In this case, the image to be stitched I m and I n Pairwise registration, if there is a point (x m ,y m), (x, y), then these two points must correspond to the same point in the target stitched image. Based on this, the target transformation matrix from the image to be stitched to the reference image can be determined as follows: according to the second overlapping area between the images to be stitched, obtain the third coordinates of multiple feature pixel points corresponding to each image to be stitched on the second overlapping area; according to the first coordinate, the second coordinate and the third coordinate, calculate the target transformation matrix of each image to be stitched and the reference image.

[0133] Specifically, the target transformation matrix of each image to be stitched and the reference image can be calculated using the following formula:

[0134] Among them, T m is the mth image to be stitched (i.e., image I m ) to the target transformation matrix of the reference image, T n is the nth image to be stitched (i.e., image I n ) to the target transformation matrix of the reference image, (x n ,y n ) is the first coordinate of a feature pixel on the nth image to be stitched, (x, y) is the second coordinate of the feature pixel on the reference image, (x m ,y m ) is the third coordinate of the feature pixel point on the mth image to be stitched.

[0135] At this point, the above two formulas are combined to solve the target transformation matrix T of each image to be stitched to the reference image. n .

[0136] In combination with the above example, the fundus image collected in step S110 is shown in FIG2 , and the image obtained after low-pass filtering is shown in FIG3 .

[0137] In this embodiment, the image obtained by preprocessing the central fundus image through low-pass filtering is defined as the reference image (as shown in Figure 4), and the images corresponding to the remaining four circumferential fundus images obtained after preprocessing are defined as the images to be stitched (as shown in Figures 5, 6, 7, and 8). Preprocessing can enhance large blood vessels to obtain more feature points, and can also obtain feature points distributed around large blood vessels, which facilitates feature extraction. Combining the original image shown in Figure 2 and the preprocessed image shown in Figure 3, it can be seen that the preprocessed image can obtain more feature points distributed around large blood vessels.

[0138] In this embodiment, a target stitched image obtained by stitching the reference image (as shown in FIG. 4 ) with four images to be stitched (as shown in FIG. 5 , FIG. 6 , FIG. 7 , and FIG. 8 ) is shown in FIG. 9 .

[0139] The present application also provides a fundus image processing device, as shown in Figure 10, which includes: an acquisition module 10, a determination module 20, a registration module 30, a feature pixel extraction module 40, a transformation matrix calculation module 50 and an image stitching module 60.

[0140] The acquisition module 10 is configured to acquire at least two fundus images captured by the OCT device, where the at least two fundus images correspond to different fundus positions of the eye to be inspected.

[0141] The determination module 20 is configured to determine a reference image and an image to be stitched other than the reference image in at least two fundus images.

[0142] The registration module 30 is configured to register each image to be stitched with the reference image, and calculate a first overlapping area between each image to be stitched and the reference image.

[0143] The feature pixel point extraction module 40 is configured to segment the key area of ​​each image to be stitched to obtain at least two image blocks, and extract the feature pixel points corresponding to each image block; wherein the key area includes the first overlapping area.

[0144] The transformation matrix calculation module 50 is configured to calculate a target transformation matrix from each image to be spliced ​​to a reference image for a plurality of characteristic pixel points corresponding to each image to be spliced.

[0145] The image stitching module 60 is configured to stitch each image to be stitched with a reference image based on a target transformation matrix corresponding to each image to be stitched, to obtain a target stitched image.

[0146] The fundus image stitching device provided in this application pre-registers each image to be stitched with a reference image, calculates the overlapping area between each pair of images, segments the overlapping area, extracts feature pixels, calculates the target transformation matrix between each image to be stitched and the reference image based on the multiple feature pixels, and then stitches each image to be stitched with the reference image based on the target transformation matrix to obtain a target stitched image. By segmenting the overlapping area and obtaining feature pixels, the uniformity of feature point extraction is improved, and optical distortion is reduced through regional registration, thereby obtaining a target stitched image with good continuity and high stitching accuracy.

[0147] In one embodiment, the reference image includes a central fundus image obtained by performing blood flow imaging on the central area of ​​the fundus, or includes an image of both the macula and the optic disc.

[0148] In one embodiment, at least two fundus images include an image corresponding to the central position of the fundus and an image corresponding to the circumferential position of the fundus; the above-mentioned determination module 20 is specifically configured to: subtract the low-pass filtering result of the image corresponding to the central position of the fundus from its low-pass filtering result to obtain a reference image, and subtract the low-pass filtering result of the image corresponding to the circumferential position of the fundus from its low-pass filtering result to obtain the image to be stitched corresponding to each image corresponding to the circumferential position of the fundus.

[0149] In one embodiment, the registration module 30 is specifically configured to: register each image to be stitched with a reference image, calculate a first offset between the image to be stitched and the reference image; and calculate a first overlapping area between each image to be stitched and the reference image based on the first offset.

[0150] In one embodiment, the feature pixel extraction module 40 is specifically configured to extract feature pixels corresponding to each image block based on feature extraction rules; wherein the feature extraction rules include one or more of the following: surrounding gradient rules, grayscale rules, brightness rules, and texture rules of feature pixels.

[0151] In one embodiment, the transformation matrix calculation module 50 is specifically configured to: obtain the first coordinates of multiple feature pixel points corresponding to each image to be stitched in the image to be stitched and the second coordinates of multiple feature pixel points in the reference image; and calculate the target transformation matrix from each image to be stitched to the reference image based on the first coordinates and the second coordinates.

[0152] In one embodiment, the transformation matrix calculation module 50 is further configured to calculate the target transformation matrix according to the following formula:

[0153] Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel point on the nth image to be stitched, and (x, y) is the second coordinate of the feature pixel point on the reference image.

[0154] In one embodiment, the image stitching module 60 is specifically configured as follows: for each image to be stitched, each feature pixel in the image to be stitched is mapped to a predicted pixel corresponding to the target stitched image according to the corresponding target transformation matrix; for each predicted pixel, a feature pixel associated with the predicted pixel in each image to be stitched is determined; and the pixel value of each feature pixel associated with the predicted pixel is weighted averaged to calculate the target pixel value to obtain the target stitched image.

[0155] In one embodiment, the image stitching module 60 is further configured to calculate the target pixel value of each predicted pixel point according to the following formula to obtain a target stitched image:

[0156] Among them, dst is the target pixel value of the predicted pixel in the target spliced ​​image, src n is the pixel value of the feature pixel associated with the predicted pixel on the nth image to be stitched, w n src n The corresponding impact factor.

[0157] In one embodiment, the above-mentioned device also includes an image adjustment module, which is configured to: calculate the overlapping area of ​​each image to be stitched and the reference image according to the target transformation matrix for each image to be stitched; perform histogram matching on the image to be stitched and the reference image in the overlapping area, and adjust the image to be stitched according to the result of the histogram matching.

[0158] In one embodiment, the image adjustment module is specifically configured to adjust the brightness and / or contrast of the image to be stitched according to the result of the histogram matching, so that the image to be stitched is consistent with the reference image in brightness and / or contrast.

[0159] In one embodiment, there are at least two images to be stitched together; the registration module 30 is further configured to calculate a second overlapping area between the images to be stitched together according to a first offset corresponding to each image to be stitched together.

[0160] In one embodiment, the registration module 30 is further configured to: determine a first offset between the first image to be stitched and the reference image, and a second offset between the second image to be stitched and the reference image; and calculate a second overlapping area between the first image to be stitched and the second image to be stitched based on the second offset and the first offset.

[0161] In one embodiment, the feature pixel extraction module 40 is further configured to: segment the first overlapping area and the second overlapping area corresponding to each image to be stitched to obtain a plurality of image blocks.

[0162] In one embodiment, the transformation matrix calculation module 50 is further configured to: obtain the third coordinates of a plurality of feature pixel points corresponding to each image to be stitched on the second overlapping area between the images to be stitched; and calculate the target transformation matrix from each image to be stitched to the reference image based on the first coordinate, the second coordinate, and the third coordinate.

[0163] In one embodiment, the transformation matrix calculation module 50 is further configured to calculate the target transformation matrix of each image to be stitched and the reference image using the following formula:

[0164] Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, T m is the target transformation matrix from the mth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a feature pixel on the nth image to be stitched, (x, y) is the second coordinate of the feature pixel on the reference image, (x m ,y m ) is the third coordinate of the feature pixel point on the mth image to be stitched.

[0165] The present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the above method.

[0166] The present application also provides a computer-readable storage medium, in which a computer program is stored. A processor executes the computer program to implement the above method.

[0167] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0168] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In addition, in the description of the embodiments of this application, unless otherwise specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0170] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0171] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of this application and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0172] Finally, it should be noted that the above embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technical field can still modify the technical solutions described in the above embodiments within the technical scope disclosed in the present application, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. Industrial Applicability

[0173] By applying the technical solution of the present application, the uniformity of feature point extraction is improved and optical distortion is reduced, thereby obtaining a target stitching image with good continuity and high stitching accuracy.

Claims

1. A fundus image stitching method, characterized in that: The method comprises: Acquire at least two fundus images acquired by an OCT device, wherein the at least two fundus images correspond to different fundus positions of the eye to be examined; Determining a reference image and an image to be stitched other than the reference image in the at least two fundus images; Registering each of the images to be stitched with the reference image respectively, and calculating a first overlapping area between each of the images to be stitched and the reference image; Segment the key area of ​​each of the images to be stitched to obtain at least two image blocks, and extract characteristic pixel points corresponding to each of the image blocks; wherein the key area includes the first overlapping area; For each of the plurality of characteristic pixel points corresponding to the image to be stitched, calculating a target transformation matrix from each of the image to be stitched to the reference image; Based on the target transformation matrix corresponding to each of the images to be stitched, each of the images to be stitched is stitched with the reference image to obtain a target stitched image.

2. The method according to claim 1, characterized in that The reference image includes a fundus center image obtained by performing blood flow imaging on the fundus center area, or includes images of the macula and the optic disc at the same time.

3. The method according to claim 1 or 2, characterized in that: The at least two fundus images include an image corresponding to the center position of the fundus and an image corresponding to the peripheral position of the fundus; The step of determining a reference image and an image to be stitched other than the reference image in the at least two fundus images comprises: The reference image is obtained by subtracting the low-pass filtering result from the image corresponding to the central position of the fundus, and the image corresponding to the circumferential position of the fundus is subtracted from the low-pass filtering result to obtain the image to be stitched corresponding to each image corresponding to the circumferential position of the fundus.

4. The method according to any one of claims 1 to 3, characterized in that: The step of registering each of the images to be stitched with the reference image and calculating a first overlapping area between each of the images to be stitched and the reference image comprises: Registering each of the images to be stitched with the reference image, and calculating a first offset between the image to be stitched and the reference image; Based on the first offset, a first overlapping area between each of the images to be stitched and the reference image is calculated.

5. The method according to any one of claims 1 to 4, characterized in that: The step of extracting characteristic pixel points corresponding to each of the image blocks comprises: The feature pixel points corresponding to each of the image blocks are extracted based on feature extraction rules; wherein the feature extraction rules include one or more of the following: surrounding gradient rules, grayscale rules, brightness rules, and texture rules of the feature pixel points.

6. The method according to claim 4, characterized in that The step of calculating the target transformation matrix from each of the images to be stitched to the reference image for the plurality of characteristic pixel points corresponding to each of the images to be stitched comprises: Acquire the first coordinates of the plurality of characteristic pixel points corresponding to each of the images to be stitched in the image to be stitched and the second coordinates of the plurality of characteristic pixel points in the reference image; A target transformation matrix from each of the images to be stitched to the reference image is calculated according to the first coordinates and the second coordinates.

7. The method according to claim 6, characterized in that The step of calculating a target transformation matrix from each of the images to be stitched to the reference image according to the first coordinates and the second coordinates includes: The target transformation matrix is ​​calculated according to the following formula: Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a certain characteristic pixel point on the nth image to be stitched, and (x, y) is the second coordinate of the characteristic pixel point on the reference image.

8. The method according to any one of claims 1 to 7, characterized in that: The step of splicing each of the images to be spliced ​​with the reference image based on the target transformation matrix corresponding to each of the images to be spliced ​​to obtain a target spliced ​​image includes: For each of the images to be stitched, calculating, according to the corresponding target transformation matrix, the predicted pixel points corresponding to the target stitched image mapped to each of the feature pixel points in the image to be stitched; For each of the predicted pixel points, determining a characteristic pixel point associated with the predicted pixel point in each of the images to be stitched; The pixel values ​​of each characteristic pixel point associated with the predicted pixel point are weighted averaged to calculate the target pixel value and obtain the target spliced ​​image.

9. The method according to claim 8, characterized in that The step of taking a weighted average of the pixel values ​​of each feature pixel point associated with the predicted pixel point, calculating the target pixel value, and obtaining the target spliced ​​image comprises: The target pixel value of each predicted pixel is calculated according to the following formula to obtain the target spliced ​​image: Wherein, dst is the target pixel value of the predicted pixel point in the target spliced ​​image, src n is the pixel value of the characteristic pixel associated with the predicted pixel on the nth image to be stitched, w n For src n The corresponding impact factor.

10. The method according to any one of claims 1 to 9, characterized in that: Before the step of splicing each of the images to be spliced ​​with the reference image based on the target transformation matrix corresponding to each of the images to be spliced ​​to obtain a target spliced ​​image, the method further includes: For each of the images to be stitched, calculating an overlapping area between each of the images to be stitched and the reference image according to the target transformation matrix; Histogram matching is performed on the image to be spliced ​​and the reference image in the overlapping area, and the image to be spliced ​​is adjusted according to the result of the histogram matching.

11. The method according to claim 10, characterized in that The step of adjusting the images to be stitched according to the result of the histogram matching comprises: The brightness and / or contrast of the image to be stitched is adjusted according to the result of the histogram matching, so that the image to be stitched is consistent with the reference image in brightness and / or contrast.

12. The method according to claim 6, characterized in that There are at least two images to be stitched; After the steps of registering each of the images to be stitched with the reference image and calculating the first overlapping area between each of the images to be stitched and the reference image, the method further includes: A second overlapping area between the images to be stitched is calculated according to the first offset corresponding to each of the images to be stitched.

13. The method according to claim 12, characterized in that The step of calculating the second overlapping area between the images to be stitched according to the first offset corresponding to each of the images to be stitched comprises: Determine a first offset between the first image to be stitched and the reference image, and a second offset between the second image to be stitched and the reference image; A second overlapping area between the first image to be stitched and the second image to be stitched is calculated based on the second offset and the first offset.

14. The method according to claim 12 or 13, characterized in that The step of segmenting the key area of ​​each image to be stitched to obtain at least two image blocks includes: The first overlapping area and the second overlapping area corresponding to each of the images to be stitched are divided to obtain a plurality of image blocks.

15. The method according to any one of claims 12 to 14, characterized in that: The step of calculating the target transformation matrix from each of the images to be stitched to the reference image for the plurality of characteristic pixel points corresponding to each of the images to be stitched further includes: According to the second overlapping area between the images to be stitched, obtaining third coordinates of a plurality of the characteristic pixel points corresponding to each of the images to be stitched on the second overlapping area; A target transformation matrix from each of the images to be stitched to the reference image is calculated according to the first coordinates, the second coordinates, and the third coordinates.

16. The method according to claim 15, characterized in that The step of calculating a target transformation matrix from each of the images to be stitched to the reference image according to the first coordinates, the second coordinates, and the third coordinates includes: The target transformation matrix of each image to be stitched and the reference image is calculated by the following formula: Among them, T n is the target transformation matrix from the nth image to be stitched to the reference image, T m is the target transformation matrix from the mth image to be stitched to the reference image, (x n ,y n ) is the first coordinate of a certain characteristic pixel point on the nth image to be stitched, (x, y) is the second coordinate of the characteristic pixel point on the reference image, (x m ,y m ) is the third coordinate of the feature pixel point on the mth image to be stitched.

17. A fundus image processing device, characterized in that: The device comprises: An acquisition module configured to acquire at least two fundus images acquired by an OCT device, wherein the at least two fundus images correspond to different fundus positions of the eye to be inspected; A determination module, configured to determine a reference image and an image to be stitched other than the reference image in the at least two fundus images; A registration module, configured to register each of the images to be stitched with the reference image, and calculate a first overlapping area between each of the images to be stitched and the reference image; a feature pixel point extraction module, configured to segment the key area of ​​each of the images to be stitched to obtain at least two image blocks, and extract feature pixel points corresponding to each of the image blocks; wherein the key area includes the first overlapping area; A transformation matrix calculation module is configured to calculate a target transformation matrix from each of the images to be spliced ​​to the reference image for a plurality of the characteristic pixel points corresponding to each of the images to be spliced; The image stitching module is configured to stitch each of the images to be stitched with the reference image based on a target transformation matrix corresponding to each of the images to be stitched, so as to obtain a target stitched image.

18. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images