A histogram matching and weighting-based retinal fundus image splicing fusion method

Abstract

The application discloses a kind of based on histogram matching and weighted retinal fundus image splicing fusion method, first the retinal fundus image of different regions of the same mode obtained is histogram matched, for reducing the brightness and contrast difference between image, then the weight matrix and affine matrix of image are sequentially obtained, and the histogram matched image and weight matrix are carried out affine transformation, finally based on the image and weight matrix after affine transformation sequentially carry out the splicing fusion of image.The scheme of the application, when histogram matching is carried out, the pixel distribution histogram of all images is counted as the target of histogram matching, reduces the brightness contrast difference between image, while avoiding the image distortion caused by using specific target;While using weight control the value of pixel when fusion image, using the way of segmentation carries out weighted summation, effectively reduces the difference at image joint, and the image fusion effect is good.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of fundus image fusion, and particularly relates to a retinal fundus image splicing and fusion method based on histogram matching and weighting. BACKGROUND

[0002] Retinal fundus images can be obtained by various imaging methods, such as color fundus (CF), infrared scanning (IR), fluorescein angiography (FA), optical coherence tomography (OCT), etc. However, due to the influence of imaging range, each image can only show local content, and increasing the imaging range of the device usually requires relatively expensive costs, which makes the splicing and fusion of fundus images a reasonable method to increase the display area of the image. Doctors can splice and fuse different positions of the same fundus image taken multiple times to obtain a complete area of the fundus image to meet the needs of clinical diagnosis and treatment.

[0003] The existing fundus image splicing technology often ignores the importance of image fusion and only uses simple weighted fusion or maximum pixel value to perform fusion. Due to the influence of light, tissue, etc. on fundus image shooting, the fundus images obtained by shooting from different angles are prone to have obvious differences in brightness and contrast. The above fusion methods will cause obvious differences in the spliced image, affecting the visual perception and clinical diagnosis. SUMMARY

[0004] In view of the above problems, the purpose of the present application is to provide a retinal fundus image splicing and fusion method based on histogram matching and weighting:

[0005] The specific technical scheme to achieve the purpose of the present application is:

[0006] A retinal fundus image splicing and fusion method based on histogram matching and weighting, comprising the following steps:

[0007] Step 1, obtaining retinal fundus images of different regions of the same modality;

[0008] Step 2, performing histogram matching on the images to reduce the brightness and contrast differences between the images;

[0009] Step 3, calculating the weight of each pixel according to the pixel distance of the image pixel and the center pixel of the image;

[0010] Step 4, performing image registration by using the feature point pair method to determine the affine matrix;

[0011] Step 5, performing affine transformation on the histogram-matched image and the weight matrix;

[0012] Step 6, sequentially performing image splicing and fusion based on the affine-transformed image and the weight matrix.

[0013] Compared with the prior art, the present application has the beneficial effects that:

[0014] (1) The scheme of the present application, when performing histogram matching, counts the pixel distribution histogram of all images as the target of histogram matching, reduces the brightness contrast difference between images, and avoids image distortion caused by using a specific target;

[0015] (2) The scheme of the present application uses the value of the pixel of the fused image when using weight control, adopts a segmented way to perform weighted summation, effectively reduces the difference at the image joint, and at the same time, the pixels of a large area are directly taken from the original image, effectively avoiding the ghosting of the fused image caused by registration error. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a step path diagram of the retinal fundus image splicing and fusion method based on histogram matching and weighting of the present application.

[0017] Figure 2 It is a retinal angiogram in the embodiment of the present application.

[0018] Figure 3 It is the pixel distribution histogram of the image I1 in the embodiment of the present application.

[0019] Figure 4 It is the overall pixel distribution histogram in the embodiment of the present application.

[0020] Figure 5 It is the pixel distribution histogram of the image I1 after histogram matching in the embodiment of the present application.

[0021] Figure 6 It is the retinal fluorescence angiogram after histogram matching in the embodiment of the present application.

[0022] Figure 7 It is the weight template image constructed in the embodiment of the present application.

[0023] Figure 8 It is a schematic diagram of the retinal fluorescence angiogram splicing and fusion result in the embodiment of the present application.

[0024] Figure 9 It is a group of retinal color fundus images in the embodiment of the present application.

[0025] Figure 10 It is a schematic diagram of the retinal color fundus image splicing and fusion result in the embodiment of the present application. DETAILED DESCRIPTION

[0026] A method for stitching and fusing retinal fundus images based on histogram matching and weighting includes the following steps:

[0027] Step 1: Acquire retinal fundus images of different regions of the same modality;

[0028] Step 2: Perform histogram matching on the images to reduce differences in brightness and contrast between images. Specifically:

[0029] Step 2-1: Denote the acquired n images as I0, I1...I n-1 If the obtained image is grayscale, proceed to step 2-2; otherwise, proceed to step 2-3.

[0030] Step 2-2: Perform histogram matching on the grayscale image, specifically as follows:

[0031] Step 2-2-1: Calculate the pixel distribution histogram for each grayscale image within the range [a, b], denoted as h0, h1...h n-1 Where a>0 and the value is extremely small, and b<255 and the value is extremely large; the number of extremely small pixel values ​​is not included in the count to avoid the influence of the black border of the image; the number of extremely large pixel values ​​is not included in the count to avoid the influence of overexposed pixels.

[0032] Step 2-2-2: Sum the n histograms pixel by pixel to obtain the overall pixel distribution histogram h of the n images. tot :

[0033]

[0034] Step 2-2-3: Calculate the cumulative distribution function (CDF) for the total n+1 histograms obtained in steps 2-2-1 and 2-2-2.

[0035]

[0036] Step 2-2-4: Based on the cumulative distribution function (CDF) values ​​of the n images, compare them with the cumulative distribution function (CDF) value of the overall pixel distribution histogram. tot Histogram matching is performed sequentially. For pixels not included in the histogram, their original pixel size is retained, resulting in n matched images I0, I1...I n-1 Complete histogram matching of grayscale images of the retina and fundus.

[0037] Steps 2-3: Perform histogram matching for the color image, specifically:

[0038] Perform the following steps for each of the three channels of n color images:

[0039] Step 2-3-1, count the pixel distribution histogram of the current channel image of each color image in the range of [a, b], denoted as h0, h1...h n-1 , where a > 0, b < 255;

[0040] Step 2-3-2, sum the n histograms pixel by pixel to obtain the overall pixel distribution histogram h of the n images of the current channel tot :

[0041]

[0042] Step 2-3-3, calculate the cumulative distribution function CDF of the total n+1 histograms obtained in step 2-3-1 and step 2-3-2:

[0043]

[0044] Step 2-3-4, according to the cumulative distribution function CDF value of the n images of the current channel and the cumulative distribution function value CDF tot of the overall pixel distribution histogram, perform histogram matching in turn, and for the pixels not counted in the histogram, keep their original pixel size and update the current channel of the color image I0, I1...I n-1 , complete the histogram matching of the current channel of the retinal fundus color image.

[0045] When the three channels of the color image are completed, the histogram matching of the color image is completed.

[0046] Step 3, calculate the weight of each pixel according to the pixel distance of the image pixel and the center of the image, specifically:

[0047] Let the current pixel coordinates be (y p , x p ), the image width be w, the image height be h, and the image center be (h / 2, w / 2). Then the pixel weight is:

[0048]

[0049] Where distance dis represents the Euclidean distance between the pixel coordinates and the image center coordinates, and mask represents whether the pixel is a meaningless border. False indicates that the pixel is part of the image, and True indicates that the pixel is a meaningless border.

[0050] Obtain the weight of each pixel for the n images in turn to obtain the pixel weight matrix W0, W1...W n-1 of the n images.

[0051] Where W iA pixel weight matrix representing the i-th image.

[0052] From the above formula, our weight matrix can be regarded as a gray image with pixel size in the range of [0-255]. In order to simplify the calculation process, a weight matrix can be pre-generated and stored as a gray image in the actual operation process. When needed, we only need to read the weight template and adjust it to the current image size through interpolation.

[0053] Step 4, image registration is performed using the feature point pair method to determine the affine matrix, specifically:

[0054] First, the feature points of the image after histogram matching are extracted, and any one image is selected as the main image. The remaining images are matched with the main image for feature points to obtain the affine matrix. Here, feature point extraction can be performed through a deep network or other extraction methods.

[0055] Step 5, affine transformation is performed on the histogram matched image and the weight matrix, specifically:

[0056] Based on the affine matrix obtained in step 4, the histogram matched image and its weight matrix are subjected to affine transformation to obtain the transformed image I'0, I'1... I' n-1 and the transformed weight matrix W'0, W'1... W' n-1 .

[0057] Step 6, based on the affine transformed image and weight matrix, the image is sequentially spliced and fused, specifically:

[0058] Step 6-1, initialize a blank canvas I c with a value of zero and a weight matrix W c ;

[0059] Step 6-2, fuse the affine transformed image with the canvas:

[0060] For image I' i , i∈{0,1,...n-1}, according to the weight matrix, the canvas I c and the weight matrix W c are updated as follows:

[0061]

[0062] Where T is a set threshold value;

[0063] Step 6-3, update the canvas with the image and weight matrix one by one until all images and the canvas are fused to obtain the final image fusion result.

[0064] The application will be further described in connection with the following examples.

[0065] Examples

[0066] In connection with Figure 1 A histogram matching and weighting-based retinal fundus image stitching and fusion method, comprising the following steps:

[0067] Step 1, obtaining retinal fundus images of different regions of the same modality, as shown in Figure 2 ;

[0068] Step 2, histogram matching of the images is performed to reduce the brightness and contrast differences between the images, specifically:

[0069] Step 2-1, the n images obtained are denoted as I0, I1...In, where n is the number of images obtained; n-1 If the obtained images are grayscale images, step 2-2 is performed, otherwise step 2-3 is performed;

[0070] In this embodiment, the obtained images are grayscale images, so step 2-2 is performed;

[0071] Step 2-2, histogram matching is completed for grayscale images, specifically:

[0072] Step 2-2-1, the pixel distribution histogram of each grayscale image in the range of [3, 250] is counted, denoted as h0, h1...hn, where n is the number of images obtained; n-1 The number of extreme pixel values (in this embodiment, 0-2) is not counted to avoid the influence of the black border of the image, and the number of extreme pixel values (in this embodiment, 251-255) is not counted to avoid the influence of overexposed pixels;

[0073] In this embodiment, the pixel distribution histogram of image I1 is shown in Figure 3 ;

[0074] Step 2-2-2, the n histograms are summed pixel by pixel to obtain the overall pixel distribution histogram h of the n images: tot

[0075]

[0076] The overall pixel distribution histogram of the n images in this embodiment is shown in Figure 4 ;

[0077] Step 2-2-3, the cumulative distribution function CDF is calculated for the total of n+1 histograms obtained in steps 2-2-1 and 2-2-2:

[0078] ​

[0079] Step 2-2-4: Based on the cumulative distribution function (CDF) values ​​of the n images, compare them with the cumulative distribution function (CDF) value of the overall pixel distribution histogram. tot Histogram matching is performed sequentially. For pixels not included in the histogram, their original pixel size is retained, resulting in n matched images I0, I1...I n-1 Complete histogram matching of grayscale images of the retina and fundus.

[0080] In this embodiment, the pixel distribution histogram of I1 after matching is as follows: Figure 5 As shown;

[0081] In this embodiment, the retinal fluorescein angiography image after histogram matching is as follows: Figure 6 As shown, the differences in brightness and contrast of the image are significantly reduced.

[0082] Steps 2-3: Perform histogram matching for the color image, specifically:

[0083] Perform the following steps for each of the three channels of n color images:

[0084] Step 2-3-1: Calculate the pixel distribution histogram for each color image, where the pixel size of the current channel image is within the range [a, b], and denote it as h0, h1...h n-1 Where a>0, b<255;

[0085] Step 2-3-2: Sum the n histograms pixel by pixel to obtain the overall pixel distribution histogram h of the n images in the current channel. tot :

[0086]

[0087] Step 2-3-3: Calculate the cumulative distribution function (CDF) for the total n+1 histograms obtained in Steps 2-3-1 and 2-3-2.

[0088]

[0089] Steps 2-3-4: Based on the cumulative distribution function (CDF) values ​​of the n images in the current channel, compare them with the cumulative distribution function (CDF) values ​​of the overall pixel distribution histogram. tot Histogram matching is performed sequentially. For pixels not included in the histogram, their original pixel size is retained, and the I0, I1...I... pixels of the color image are updated. n-1 The current channel is used to complete the histogram matching of the current channel of the retinal fundus color image.

[0090] Once histogram matching is completed for all three channels of a color image, the histogram matching for the color image is complete.

[0091] Step 3: Calculate the weight of each pixel based on the pixel distance between the image pixel and the image center, specifically as follows:

[0092] Let the current pixel coordinates be (y p ,x p Given an image with width w and height h, and its center at (h / 2, w / 2), the pixel weights are:

[0093]

[0094] Where, distance dis represents the Euclidean distance between the pixel coordinates and the center coordinates of the image, mask indicates whether the pixel is a meaningless border, False indicates that the pixel is a valid part of the image, and True indicates that the pixel is a meaningless border;

[0095] The weight of each pixel in each of the n images is calculated sequentially, resulting in a pixel weight matrix W0, W1...W for the n images. n-1 ;

[0096] Among them, W i Let represent the pixel weight matrix of the i-th image.

[0097] As shown in the above formula, our weight matrix can be considered as a grayscale image with pixel sizes in the range of [0-255]. To simplify the calculation process, in practice, a weight matrix of size 512×512 can be pre-generated and stored as a weight template in a grayscale image. When needed, we only need to read the weight template and adjust it to the current image size through interpolation. The weight matrix template in this embodiment is as follows: Figure 7 As shown, at the beginning of the program we will read it into memory. After reading the image, the corresponding weight matrix can be obtained according to the image size and the mask, thus avoiding repeated calculation of the weights.

[0098] Step 4: Perform image registration using feature point pairs to determine the affine matrix, specifically as follows:

[0099] First, feature points are extracted from the images after histogram matching. Then, any image is selected as the main image, and the feature points of the remaining images are matched with the main image to obtain the affine matrix. Feature point extraction can be performed using deep networks or other extraction methods.

[0100] Step 5: Perform an affine transformation on the histogram-matched image and the weight matrix, specifically:

[0101] Based on the affine matrix obtained in step 4, an affine transformation is performed on the image after histogram matching and its weight matrix to obtain the transformed image I′0, I′1...I′ n-1and the transformed weight matrix W'0, W'1...W' n-1 .

[0102] Step 6, based on the image after the affine transformation and the weight matrix, the image is sequentially spliced and fused, specifically;

[0103] Step 6-1, initialize a blank canvas I with a value of zero c and a weight matrix W with a value of zero c , wherein the size of the blank canvas I c can be calculated by the affine matrix in step 4, and the size calculated by the affine matrix is the optimal size, which can save computing resources;

[0104] Step 6-2, fuse the image after the affine transformation with the canvas:

[0105] for image I i ′ , i∈{0,1,...n-1}, according to the weight matrix, the canvas I c and the weight matrix W c are updated as follows:

[0106]

[0107] wherein T is a set threshold value;

[0108] Step 6-3, update the canvas with the image and the weight matrix one by one until all images and the canvas are fused to obtain the final image fusion result.

[0109] The retinal fluorescence angiography image after splicing and fusion by the method in the embodiment is shown in Figure 8 According to our segmented weighted fusion method, the images are fused into the canvas one by one, and it can be seen that the overall brightness and contrast of our fusion result is consistent, and the difference at the image joint is not obvious, and a good fusion effect is obtained.

[0110] Figure 9 and Figure 10 show that our fusion method also has good fusion effect on color fundus images.

[0111] The above embodiments show and describe the basic principles and main features of the present application. Those skilled in the art should understand that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application.

Claims

1. A method for fusing retinal fundus images based on histogram matching and weighting, characterized in that, The method comprises the following steps: Step 1, obtaining retinal fundus images of different regions of the same modality; Step 2, performing histogram matching on the images to reduce the brightness and contrast difference between the images; Step 2-1, denote the acquired n images as I0, I1...In n-1 If the acquired image is a gray image, execute Step 2-2, otherwise execute Step 2-3; Step 2-2, performing histogram matching on the gray-scale images: Step 2-2-1, count the pixel distribution histogram of the pixel size in the range of [a, b] of each gray image, denoted as h0, h1...h n-1 where a > 0, b < 255; Step 2-2-2, summing the n histograms pixel by pixel to obtain the overall pixel distribution histogram h of the n images tot : Step 2-2-3, calculating the cumulative distribution function CDF of the total n+1 histograms obtained in steps 2-2-1 and 2-2-2: Step 2-2-4, according to the cumulative distribution function CDF value of n pictures respectively and the cumulative distribution function value CDF of the overall pixel distribution histogram tot , sequentially carrying out histogram matching, for the pixels not counted in the histogram, the original pixel size will be retained, obtaining the matched n images I0, I1...I n-1 , completing the histogram matching of the retinal fundus gray scale image; Step 2-3, performing histogram matching on the color images: Step 3, calculating the weight of each pixel according to the pixel distance of the image pixel and the center pixel of the image: Let the current pixel coordinate be (y p ,x p ), the image width be w, the image height be h, and the image center be (h / 2, w / 2). The pixel weight is: Where distance dis represents the Euclidean distance of the pixel coordinates and the center coordinates of the image, mask represents whether the pixel is a meaningless frame, False represents that the pixel is an effective part of the image, and True represents that the pixel is a meaningless frame; The pixel weight of each picture is sequentially obtained, and the pixel weight matrix W0, W1...Wn of the n pictures is obtained n-1 ; where W i represents the pixel weight matrix of the i-th picture; Step 4, performing image registration using feature point pairs to determine the affine matrix; Step 5, performing affine transformation on the histogram-matched images and the weight matrix; Step 6, sequentially performing image stitching and fusion based on the affine-transformed images and the weight matrix.

2. The method for retinal fundus image mosaic fusion based on histogram matching and weighting according to claim 1, characterized in that, The histogram matching for color images in step 2-3 is specifically: The following steps are performed on the three channels of the n color images: Step 2-3-1, count the pixel distribution histogram of the current channel image of each color image in the range of [a, b], denoted as h0, h1...h n-1 where a > 0, b < 255; Step 2-3-2, summing up the n histograms pixel by pixel to obtain the overall pixel distribution histogram h of the n images of the current channel tot : Step 2-3-3, calculating the cumulative distribution function CDF of the total n+1 histograms obtained in steps 2-3-1 and 2-3-2: Step 2-3-4, according to the cumulative distribution function CDF value of the current channel of n pictures and the cumulative distribution function value CDF of the overall pixel distribution histogram tot , sequentially perform histogram matching, for the pixels not counted in the histogram, the original pixel size will be retained, and the I0, I1...I n-1 of the current channel of the color image is updated, and the histogram matching of the current channel of the retinal fundus color image is completed.

3. The method for histogram matching and weighted based retinal fundus image mosaic fusion according to claim 1, characterized in that, The determination of the affine matrix in step 4 is specifically: First, feature points are extracted from the histogram-matched images, and any one image is selected as the main image, and the remaining images are matched with the main image to obtain the affine matrix.

4. The method for histogram matching and weighted based fusion of retinal fundus images mosaicking according to claim 1, wherein, The affine transformation of the histogram-matched images and the weight matrix in step 5 is specifically: Based on the affine matrix acquired in step 4, the image after histogram matching processing and its weight matrix are subjected to affine transformation to obtain the transformed image I'0, I'1...I' n-1 and the transformed weight matrix W'0, W'1...W' n-1 .

5. The method for stitching fusion of retinal fundus images based on histogram matching and weighting according to claim 4, characterized in that, The image stitching and fusion in step 6 is specifically: Step 6 - 1, initialize a blank canvas I with a value of zero c and a weight matrix W with a value of zero c ; Step 6-2, fusing the affine-transformed images with the canvas: For image I' i , i∈{0,1,...n-1}, according to the weight matrix, the canvas I c and the weight matrix W c are updated in turn as follows: Where T is a set threshold value; Step 6-3, updating the canvas with the images and the weight matrix one by one until all the images are fused with the canvas to obtain the final image fusion result.

Images