An image fusion method based on dictionary model

Abstract

A dictionary-based image fusion method includes obtaining image features of each coordinate point of the image to be fused in three band images, namely, first band features, second band features, and third band features; for each coordinate point, indexing and calculating a first channel index value in a first dictionary model based on the first band features; indexing and calculating a second channel index value in a second dictionary model based on the second band features and the first channel features; and indexing and calculating a third channel index value in a third dictionary model based on the third band features and the first channel features; and superimposing the first channel index value, the first channel index value, and the third channel index value of each coordinate point to obtain a fused true color image. The present invention establishes a mapping ratio relationship between the three-channel and three-band image features and utilizes a hierarchical feature indexing approach to rapidly restore a true color image.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to an image fusion method based on a dictionary model. Background Art

[0002] Medical imaging technology has rapidly developed and has become indispensable in medical diagnosis. Since the advent of the digital imaging era, the generation of massive amounts of data has opened up new possibilities for the future development of medical imaging. Therefore, how to further analyze and mine medical imaging big data, how to extract valuable information from high-dimensional medical image data, and how to closely integrate the development of modern medical imaging with precision medicine have become important issues in the future development of medical imaging.

[0003] Due to the varying degrees of advancement in medical equipment, some devices are capable of outputting true-color images, while others can only output images in three or more wavelengths. True-color images offer a high degree of color saturation that accurately reflects subtle changes in human tissue structure and more closely reflects the subjective perception of color by the human eye. Simply superimposing these three wavelengths results in a dim, low-saturation image, hindering medical staff from observing lesions or planning surgeries. Existing methods employ deep learning to fuse these three wavelengths, but this approach is complex, slow, and requires high-quality equipment. Therefore, a technology is needed that can rapidly fuse these three wavelengths to produce true-color images. Summary of the Invention

[0004] To this end, an embodiment of the present invention provides an image fusion method based on a dictionary model to solve the problems in the prior art of poor image effect and low color saturation caused by simple superposition of three-band images and low efficiency of existing true color image fusion methods.

[0005] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] An image fusion method based on a dictionary model includes the following steps:

[0007] Step 1: Obtain the image features of each coordinate point of the image to be fused in the three band images, which are the first band features, the second band features and the third band features respectively.

[0008] Step 2: For each coordinate point, obtain a first mapping ratio by indexing in the first dictionary model based on the first band feature, and calculate a first channel index value based on the first band feature and the first mapping ratio. Obtain the first channel feature based on the first channel index value, obtain a second mapping ratio by indexing in the second dictionary model based on the second band feature and the first channel feature, and calculate a second channel index value based on the second band feature and the second mapping ratio; obtain a third mapping ratio by indexing in the third dictionary model based on the third band feature and the first channel feature, and calculate a third channel index value based on the third band feature and the third mapping ratio.

[0009] Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image.

[0010] Furthermore, for a single coordinate point, a local area around the coordinate point is selected based on a preset size, and the image feature is a statistical value of the grayscale of the local area.

[0011] Furthermore, in step 2, if there is a value greater than 255 among the first channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255; or, if there is a value greater than 255 among the second channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255; or, if there is a value greater than 255 among the third channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255.

[0012] Furthermore, the method for constructing the first dictionary model, the second dictionary model, and the third dictionary model in step 2 includes the following steps:

[0013] Step 2-1, obtaining the image features of each coordinate point of the sample image in the three band images, namely the first band feature, the second band feature, and the third band feature; obtaining the image features of the three channels of each coordinate point, namely the first channel feature, the second channel feature, and the third channel feature;

[0014] Step 2-2: For a single coordinate point, calculate the first mapping ratio of the first band feature to the first channel feature, use the first band feature as an index condition, establish an association relationship between the index condition and the first mapping ratio, and count the association relationship for each coordinate point to obtain a first dictionary model. For a single coordinate point, calculate the second mapping ratio of the second channel feature to the first channel feature, use the second band feature and the first channel feature as index conditions, establish an association relationship between the index condition and the second mapping ratio, and count the association relationship for each coordinate point to obtain a second dictionary model. For a single coordinate point, calculate the third mapping ratio of the third channel feature to the first channel feature, use the third band feature and the first channel feature as index conditions, establish an association relationship between the index condition and the third mapping ratio, and count the association relationship for each coordinate point to obtain a third dictionary model.

[0015] Furthermore, before step 2-1, the method further includes obtaining a band image and a true color image collected for the same target as a sample image; the image features of the three channels of each coordinate point obtained in step 2-1 are the image features of each coordinate point in the three channels of the true color image, which are the first channel features, the second channel features and the third channel features respectively.

[0016] Furthermore, in step 2-2, the statistical association of the coordinate points to obtain the first dictionary model specifically comprises: when the index conditions of the first mapping ratios of multiple coordinate points are the same, the average of the multiple first mapping ratios is used as the index result of the index condition. And / or, the statistical association of the coordinate points to obtain the second dictionary model specifically comprises: when the index conditions of the second mapping ratios of multiple coordinate points are the same, the average of the multiple second mapping ratios is used as the index result of the index condition. And / or, the statistical association of the coordinate points to obtain the third dictionary model specifically comprises: when the index conditions of the third mapping ratios of multiple coordinate points are the same, the average of the multiple third mapping ratios is used as the index result of the index condition.

[0017] Furthermore, if more than three bands of images are obtained in step 2-1, the band images with similar bands are merged, and finally three images are retained, and image features are extracted from each of them.

[0018] Furthermore, the image features of the three channels of each coordinate point are obtained in the step 2-1, specifically including: calculating the descriptor of each coordinate point in the selected band image; for a single coordinate point, obtaining the index results of the first channel feature, the second channel feature and the third channel feature based on the index of the descriptor in the fourth dictionary model, and using the index results as the image features of the three channels of the coordinate point.

[0019] Furthermore, the image features in the three channels of each coordinate point of the true color image in the sample image are obtained, namely the first channel feature, the second channel feature and the third channel feature, and the descriptor of each coordinate point in the selected channel is calculated; the first channel feature, the second channel feature and the third channel feature are used as index results, and the association relationship between the descriptor and the index result is established, and the association relationship of each coordinate point is counted to obtain the fourth dictionary model.

[0020] Furthermore, the method for calculating the descriptor of each coordinate point includes:

[0021] Step 3-1, calculate the centroid coordinates (x, y) of the image block centered at the coordinate point (i, j) and with a side length of m in the image according to the grayscale centroid method.

[0022] Step 3-2, calculate the vector from the coordinate point (i, j) to the center of mass coordinate (x, y), and the direction of the vector is used as the angular direction of the coordinate point (i, j).

[0023] Step 3-3, calculating the descriptor of point (i, j) according to the angle direction and the grayscale information of the image block.

[0024] The embodiments of the present invention have the following advantages:

[0025] The present invention establishes a mapping proportional relationship between the image features of three channels and three bands. Compared with the method of simply superimposing three-band images, the color of the image after fusion by this method will be close to the image sample, with higher color richness and closer to the color perceived by the human eye. In addition, this method can be used in medical imaging to more accurately observe lesions, effectively reducing the misdiagnosis rate. Its application in medical equipment is conducive to surgical planning and real-time monitoring during surgery, improving surgical accuracy, reliability and safety, and has high application value. In addition, compared with the fusion method using deep learning, the present invention has a simpler processing flow, lower equipment requirements, can be widely used in various types of medical imaging equipment, and is faster, and can meet the requirements of real-time output of color images in some special scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0027] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not intended to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in mapping proportions, or adjustment in size should still fall within the scope of the technical contents disclosed in the present invention without affecting the efficacy and objectives that can be achieved by the present invention.

[0028] Figure 1 A flowchart of an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0029] Figure 2 A flowchart of a method for using a dictionary in an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0030] Figure 3 A comparison diagram of a simple superposition image of three-band images and a restored true color image of an image to be fused in an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0031] Figure 4 A flowchart of a method for dictionary construction phase in an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0032] Figure 5 A flowchart of a method for using a dictionary in an image fusion method based on a dictionary model provided in Example 2 of the present invention;

[0033] Figure 6 for Figure 5 Flowchart of the method for calculating feature descriptors in . DETAILED DESCRIPTION

[0034] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0035] Example 1

[0036] An image fusion method based on dictionary model, such as Figure 1 As shown, the following steps are included:

[0037] Step 1: Obtain the image features of each coordinate point in the three band images of the image to be fused, which are the first band features, the second band features and the third band features.

[0038] In conjunction with the previous example, the three-band image to be fused and the three-band image used to construct the dictionary model can be of the same or different types. In this embodiment, the same type is selected, with any one of the three-band images serving as the first band image, and the remaining two serving as the second and third band images, respectively. In this embodiment, the blue light image is used as the first band image, the red light image as the second band image, and the green light image as the third band image. If more than three bands of images are acquired, images with similar bands are merged, ultimately retaining three images and extracting image features from each. Preferably, the three bands are close to the red, green, and blue bands, respectively. The merging method can be, for example, taking the grayscale mean, weighted average, or maximum value of the images of the close bands. For a single coordinate point, a local area surrounding the coordinate point is selected based on a preset size. The image feature is the statistical value of the grayscale of the local area. The statistical value can be the mean, maximum, median, etc. In this embodiment, the first band feature of the coordinate point (i, j) is the grayscale value mean blueLM(i, j) of the local area around the coordinate point (i, j) in the blue light imaging image of the image to be fused, the second band feature is the grayscale value mean redLM(i, j) of the local area around the coordinate point (i, j) in the red light imaging image of the image to be fused, and the third band feature is the grayscale value mean greenLM(i, j) of the local area around the coordinate point (i, j) in the green light imaging image of the image to be fused.

[0039] The first band feature, the second band feature, and the third band feature may be image features of the same nature or of different nature. As long as the technical solution can be used to reflect the image features at the coordinate points in the first band image, the second band image, and the third band image, it falls within the scope of protection of the present invention. The image feature may be the image feature of a single coordinate point or the image feature of a local area composed of multiple coordinate points:

[0040] The image feature of a single coordinate point may be a grayscale value (ie, pixel value) or a contrast value of a single pixel.

[0041] A local area composed of multiple coordinate points can be set to various shapes, such as rectangle, square, circle, etc. The size parameters of the local area are determined by its shape. For example, if the local area is a rectangle, the size parameters are length and width, and the selected coordinate points can be special points such as the intersection of the rectangle's diagonals, corner points, and center points of an edge. For example, if the local area is a circle, the size parameter is the radius, and the selected coordinate point is the center of the circle. The image features of the local area include one or more of the grayscale value extreme values, grayscale value mean values, contrast value extreme values, contrast value mean values, gradient value extreme values, gradient value mean values, histogram mean values, local grayscale variance, local / global grayscale variance mean values, Brenner gradient, Laplace gradient, Sobel gradient, Tenegrad gradient, SMD gradient, SMD2 gradient, Roberts gradient, energy gradient, local / global information entropy, grayscale co-occurrence matrix features, Tamura texture features, autoregressive texture model, wavelet transform features, energy spectrum function features, and LBP texture features of all pixels in the local area, where the extreme values ​​mentioned are maximum values, minimum values, and sum values, etc. In this embodiment, it is impossible to enumerate the features of a single pixel or multiple pixels. As long as the features can be used as a single coordinate point or multiple coordinate points, they fall within the protection scope of the present invention.

[0042] Taking the grayscale value mean of the local area of ​​point (i, j) in the first band image as the first band feature of point (i, j) as an example, the local area is a square with a side length of n and centered at point (i, j). The first band feature is the grayscale value mean blueLM(i, j) of the local area of ​​point (i, j) in the first band image. The formula is as follows:

[0043]

[0044] Where:

[0045] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0046] imageBLUE(p,q) is the grayscale value of the pixel at the coordinate point (p,q) in the first band image;

[0047] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0048] Taking the grayscale value mean of the local area of ​​the second band image (i, j) as the second band feature as an example, the local area is a square with a side length of n and centered at point (i, j). The second band feature is the grayscale value mean redLM(i, j) of the local area of ​​the second band image (i, j). The formula is as follows:

[0049]

[0050] Where:

[0051] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0052] imageRED(p,q) is the grayscale value of the pixel at the coordinate point (p,q) in the second band image;

[0053] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0054] Taking the grayscale value mean of the local area of ​​the point (i, j) in the third band image as the third band feature as an example, the local area is a square with a side length of n and centered at the point (i, j). The third band feature is the grayscale value mean greenLM(i, j) of the local area of ​​the point (i, j) in the third band image. The formula is as follows:

[0055]

[0056] Where:

[0057] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0058] imageGREEN(p,q) is the grayscale value of the pixel at the coordinate point (p,q) in the third band image;

[0059] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0060] Step 2: For each coordinate point, obtain the first mapping ratio by indexing in the first dictionary model based on the first band feature, and calculate the first channel index value according to the first band feature and the first mapping ratio; obtain the first channel feature according to the first channel index value; obtain the second mapping ratio by indexing in the second dictionary model based on the second band feature and the first channel feature, and calculate the second channel index value according to the second band feature and the second mapping ratio; obtain the third mapping ratio by indexing in the third dictionary model based on the third band feature and the first channel feature, and calculate the third channel index value according to the third band feature and the third mapping ratio.

[0061] like Figure 2 As shown, the following takes the coordinate point (i, j) as an example to illustrate the specific method of obtaining the first channel index value, the second channel index value, and the third channel index value of the coordinate point (i, j):

[0062] a. First channel index value

[0063] Get the first band feature blueLM(i,j) of the coordinate point (i,j) in the first band image of the image to be fused, use the first band feature blueLM(i,j) as the index condition to index in the pre-built first dictionary model, and get its corresponding first mapping ratio blueLM(i,j) / bLM(i,j) , and then calculate the first channel index value blueAfter(i,j) of the coordinate point (i,j) based on the first band feature blueLM(i,j) and the first mapping ratio. The calculation method of the first channel index value varies depending on the method of generating the first mapping ratio. In this embodiment, the calculation method of the first channel index value is to calculate the first band feature blueLM(i,j) and the first mapping ratio blueLM(i,j) / bLM(i,j) The quotient of is used to get the first channel index value blueAfter(i,j) of the coordinate point (i,j). The specific formula is as follows:

[0064] blueAfter(i,j)=blueLM(i,j) / ratioB(round(blueLM(i,j)),1));

[0065] In the formula, the round function is a rounding function, and ratioB(round(blueLM(i,j)),1)) is the value in the matrix that stores the association relationship of the first mapping ratio. If ratioB(round(blueLM(i,j)),1)) or round(blueLM(i,j)) is 0, the formula is not executed. If there are values ​​greater than 255 in the first channel features of all fused coordinate points, normalization is performed and then multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all fused coordinate points are not greater than 255, normalization can still be performed and then multiplied by 255 to uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0066] The first channel feature is obtained according to the first channel index value. Specifically, after obtaining the first channel index values ​​blueAfter of all the coordinate points of the image to be fused, for each coordinate point, the first channel index value blueAfter(i, j) is used as the grayscale value of the coordinate point (i, j) in the first channel image obtained by the index, and with reference to the calculation method of the first band feature of the aforementioned coordinate point, the first channel feature blueAfterLM(i, j) of the coordinate point is obtained, that is, the first channel feature of the point (i, j) is the grayscale value mean blueAfterLM(i, j) of the square local area with the coordinate point (i, j) as the midpoint and the side length n in the first channel image obtained by the index.

[0067] b. Second channel index value

[0068] Obtain the second band feature redLM(i,j) of coordinate point (i,j) in the second band image of the image to be fused. That is, the grayscale mean of the local area formed by a square with a side length of n centered at coordinate point (i,j) is used as the second band feature redLM(i,j) of coordinate point (i,j). Use the second band feature redLM(i,j) and the first channel feature blueAfterLM(i,j) obtained by the index calculation as index conditions to index into the second dictionary model, obtaining the second mapping ratio rLM(i,j) / bLM(i,j). Then, calculate the second channel index value redAfter(i,j) of coordinate point (i,j) based on the first channel feature blueAfterLM(i,j) of coordinate point (i,j) and the second mapping ratio. The calculation method of the second channel index value varies depending on the method for generating the second mapping ratio. In this embodiment, the calculation method of the second channel index value is to calculate the product of the first channel feature blueAfterLM(i, j) and the second mapping ratio rLM(i, j) / bLM(i, j) to obtain the second channel index value redAfter(i, j) of the coordinate point (i, j). The specific formula is as follows:

[0069] redAfter(i,j)=ratioRB(round(redLM(i,j)),round(blueAfterLM(i,j)))*blueAfterLM(i,j);

[0070] In the formula, the round function is a rounding function, and ratioRB(round(redLM(i,j)),round(blueAfterLM(i,j))) is the value in the matrix that stores the association relationship of the second mapping ratio. If any of ratioRB(round(redLM(i,j)), round(blueAfterLM(i,j))), round(redLM(i,j)), round(blueAfterLM(i,j)), and blueAfterLM(i,j) is 0, the formula is not executed. If there is a value greater than 255 in the second channel features of all the fused coordinate points, normalization is performed and then multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent the existence of erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all the fused coordinate points are not greater than 255, normalization can still be performed and then multiplied by 255 to uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0071] c. Third channel index value

[0072] Obtain the third-band feature greenLM(i,j) of coordinate point (i,j) in the third-band image of the image to be fused. Specifically, use the grayscale mean of the local area formed by a square with side length n centered at coordinate point (i,j) as the third-band feature greenLM(i,j) of coordinate point (i,j). Use the third-band feature greenLM(i,j) and the first-channel feature blueAfterLM(i,j) obtained by the index calculation as index conditions to index into the third dictionary model, obtaining the third mapping ratio gLM(i,j) / bLM(i,j). Then, calculate the third channel index value greenAfter(i,j) of coordinate point (i,j) based on the first-channel feature blueAfterLM(i,j) of coordinate point (i,j) and the third mapping ratio. The calculation method of the third channel feature varies depending on the method for generating the third mapping ratio. In this embodiment, the calculation method of the third channel feature is to calculate the product of the first channel feature blueAfterLM(i, j) and the third mapping ratio gLM(i, j) / bLM(i, j) to obtain the third channel index value greenAfter(i, j) of the coordinate point (i, j). The specific formula is as follows:

[0073] greenAfter(i,j)=ratioGB(round(greenLM(i,j)),round(blueAfterLM(i,j)))*blueA afterLM(i,j);

[0074] In the formula, the round function is a rounding function, and ratioGB(round(greenLM(i,j)),round(blueAfterLM(i,j))) is the value in the matrix that stores the association relationship of the third mapping ratio. If any of ratioGB(round(greenLM(i,j)), round(blueAfterLM(i,j))), round(greenLM(i,j)), round(blueAfterLM(i,j)), and blueAfterLM(i,j) is 0, the formula is not executed. If the third channel index values ​​of all the fused coordinate points are greater than 255, they are normalized and multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all the fused coordinate points are not greater than 255, they can still be normalized and multiplied by 255 to uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0075] Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image. Figure 3 The image on the left is the image obtained by simply superimposing the three-band images to be fused. Figure 3 The image on the right is the true color image after the three-band image is fused by this method. Figure 3 The images on the left and right show that the transformed true-color images have richer color saturation and colors that are more consistent with human perception, resulting in a more realistic image and significantly improved image quality. Furthermore, this method can be used in medical imaging to more accurately identify lesions, effectively reducing misdiagnosis rates. Its application in medical devices facilitates surgical planning and real-time monitoring during surgery, improving surgical accuracy, reliability, and safety.

[0076] The following examples illustrate the construction method of the first dictionary model, the second dictionary model, and the third dictionary model in step 2:

[0077] Step 2-1: Obtain the image features of each coordinate point of the sample image in the three band images, namely the first band feature, the second band feature, and the third band feature. Also obtain the image features of the three channels of each coordinate point, namely the first channel feature, the second channel feature, and the third channel feature.

[0078] Before step 2-1, the method further includes acquiring three band images and a true color image of the same target as sample images, and then acquiring the image features of each coordinate point in the three band images to obtain the first band feature, second band feature, and third band feature of each coordinate point. Furthermore, the method further includes acquiring the image features of each coordinate point in the three channels of the true color image, which are the first channel feature, the second channel feature, and the third channel feature. Specifically, the method includes:

[0079] In step 2-1-1, three-band images of the current image are obtained respectively, namely, a first band image, a second band image, and a third band image.

[0080] In this embodiment, the three-band image includes a red light imaging image, a blue light imaging image, and a green light imaging image. In this technology, any one of the three-band images is used as the first band image, and the remaining two images are respectively the second band image and the third band image. In this embodiment, the blue light imaging image is used as the first band image, the red light imaging image is used as the second band image, and the green light imaging image is used as the third band image. It should be emphasized that if image features of band images exceeding three bands are obtained, the image features of bands with similar bands are merged to merge the images exceeding three bands into a three-band image. The merging method may be, for example, taking the average of the similar bands.

[0081] In step 2-1-2, the first band features of each coordinate point in the current image located in the first band image, the second band features of the second band image, and the third band features of the third band image are obtained. Based on the previous example, the first band features are the image features of each coordinate point in the blue light imaging image, the second band features are the image features of each coordinate point in the red light imaging image, and the third band features are the image features of each coordinate point in the green light imaging image.

[0082] In step 2-1-3, for a true color image acquired from the same target, three-channel images of the true color image are obtained to obtain a first channel image, a second channel image, and a third channel image. In this embodiment, the three-channel images are specifically a B channel image, an R channel image, and a G channel image. In this embodiment, the B channel image is set as the first channel image, the R channel image is set as the second channel image, and the G channel image is set as the third channel image.

[0083] In step 2-1-4, the first channel feature of each coordinate point in the first channel image, the second channel feature in the second channel image, and the third channel feature in the third channel image are obtained. To facilitate subsequent calculations, the first channel feature is considered to be the same type of image feature as the first band feature, the second channel feature is considered to be the same type of image feature as the second band feature, and the third channel feature is considered to be the same type of image feature as the third band feature.

[0084] Combined with the previous example, the first band feature of point (i, j) is the grayscale value mean blueLM(i, j) of the local square area with the coordinate point (i, j) as the midpoint and the side length n in the first band image. The first channel feature of point (i, j) is the grayscale value mean bLM(i, j) of the local square area with the coordinate point (i, j) as the midpoint and the side length n in the first channel image. The formula is as follows:

[0085]

[0086] Where:

[0087] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0088] imageB(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0089] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0090] Combined with the previous example, the second band feature is the gray value mean redLM(i, j) of the local area of ​​the square with the coordinate point (i, j) as the center and the side length n in the second band image. The second channel feature is the gray value mean rLM(i, j) of the local area of ​​the coordinate point (i, j) in the second channel image. The formula is as follows:

[0091]

[0092] Where:

[0093] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0094] imageR(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0095] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0096] Combined with the previous example, the third band feature is the gray value mean greenLM(i, j) of the local area of ​​the square with the coordinate point (i, j) as the center and the side length n in the third band image. The third channel feature is the gray value mean gLM(i, j) of the local area of ​​the coordinate point (i, j) in the third channel image. Its formula is as follows:

[0097]

[0098] Where:

[0099] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate, q is the vertical coordinate;

[0100] imageG(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0101] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0102] Step 2-2: for a single coordinate point, calculate the first mapping ratio of the first band feature to the first channel feature, establish an association relationship between the first band feature and the first mapping ratio, and count the association relationship of each coordinate point to obtain a first dictionary model.

[0103] For a single coordinate point, a second mapping ratio of the second channel feature to the first channel feature is calculated, the second band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the second mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a second dictionary model;

[0104] For a single coordinate point, the third mapping ratio of the third channel feature to the first channel feature is calculated, the third band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the third mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a third dictionary model.

[0105] like Figure 4 As shown, the following describes the generation method of the first dictionary model, the second dictionary model, and the third dictionary model in detail with reference to the previous example:

[0106] 2-2-1, the first dictionary model

[0107] For a single coordinate point, calculate the first mapping ratio of the first band feature and the first channel feature, set the first band feature as the index condition of the first mapping ratio, and establish the association relationship between the first band feature and the first mapping ratio. Taking the coordinate point (i, j) as an example, calculate the first mapping ratio of the first band feature blueLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j), that is, blueLM(i,j) / bLM(i,j) , and vice versa. The first band feature blueLM(i,j) is set as the index condition of the first mapping ratio of the coordinate point (i,j), and the association relationship between the first band feature and the first mapping ratio is established. A matrix ratioB can be set to save the association relationship of the first mapping ratio of all coordinate points. Here, all coordinate points refer not only to all coordinate points in an image, but also to all coordinate points in all images used to establish the dictionary. In this embodiment, the size of the matrix ratioB is 256*1, and the formula for the first mapping ratio of the coordinate point (i,j) in the matrix ratioB is:

[0108] ratioB(round(blueLM(i,j)),1)=ratioB(round(blueLM(i,j)),1)+blueLM(i,j) / bLM(i,j);

[0109] When multiple index conditions of the same first mapping ratio appear, the formula accumulates the multiple first mapping ratios. Therefore, a matrix ratioBN is required to count and save the number of occurrences of the first band feature. In this embodiment, the size of the matrix ratioBN is 256*1. If the same first band feature appears, the statistical number is increased by 1. The formula of the matrix ratioBN is:

[0110] ratioBN((round(blueLM(i,j)),1)=ratioBN((round(blueLM(i,j)),1)+1;

[0111] The round function in the formula is the rounding function.

[0112] When multiple identical first-band features appear, the value of the matrix ratioB is divided by the value of the matrix ratioBN to obtain the mean of the first mapping ratios corresponding to the first-band features. In this embodiment, this mean is used as the index result in the first dictionary model. Those skilled in the art may also use statistical values ​​such as the maximum value or median as the index result.

[0113] 2-2-2, the second dictionary model

[0114] For a single coordinate point, the second mapping ratio of the second channel feature to the first channel feature is calculated, and the second band feature and the first channel feature are used as index conditions to establish an association relationship between the index condition and the second mapping ratio. Taking the coordinate point (i, j) as an example, the second mapping ratio of the second channel feature rLM(i, j) of the coordinate point (i, j) and the first channel feature bLM(i, j) is calculated, that is, rLM(i, j) / bLM(i, j), and vice versa; the combination of the second band feature redLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j) is set as the index condition of the second mapping ratio, and an association relationship between the index condition of the second mapping ratio and the second mapping ratio is established. A matrix ratioRB can be set to save the association relationship of the second mapping ratios of all coordinate points, where all coordinate points refer not only to all coordinate points in an image, but also to all coordinate points in all images used to establish a dictionary. In this embodiment, the size of the matrix ratioRB is 256*256, and the formula for the second mapping ratio of the coordinate point (i, j) in the matrix ratioRB is:

[0115] ratioRB(round(redLM(i,j)),round(bLM(i,j)))=ratioRB(round(redLM(i,j)),round(bLM(i,j)))+rLM(i,j) / bLM(i,j)

[0116] When multiple identical index conditions of the second mapping ratio appear, the formula accumulates the multiple second mapping ratios. Therefore, a matrix ratioBN is required to count and save the number of occurrences of the index conditions of the second mapping ratio. In this embodiment, the size of the matrix ratioBN is 256*1. If the identical index condition of the second mapping ratio appears, the statistical number is increased by 1. The formula of the matrix ratioBN is:

[0117] ratioRBN(round(redLM(i,j)),round(bLM(i,j)))=ratioRBN(round(redLM(i,j)),round(bLM(i,j)))+1;

[0118] The round function in the formula is the rounding function.

[0119] When multiple index conditions for the same second mapping ratio appear, the average of the multiple second mapping ratios can be obtained by dividing the value of the matrix ratioRB by the value of the matrix ratioRBN. In this embodiment, this average is used as the index result in the second dictionary model. Those skilled in the art may also use a statistical value such as the maximum value or median as the index result.

[0120] 2-2-3, the third dictionary model

[0121] For a single coordinate point, the third mapping ratio of the third channel feature and the first channel feature is calculated, and the third band feature and the first channel feature are used as index conditions to establish an association relationship between the index condition and the third mapping ratio. Taking the coordinate point (i, j) as an example, the third mapping ratio of the third channel feature gLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j) is calculated, that is, gLM(i, j) / bLM(i, j), and vice versa; the combination of the third band feature greenLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j) is set as the index condition of the third mapping ratio, and an association relationship between the index condition of the third mapping ratio and the third mapping ratio is established. A matrix ratioGB can be set to save the third mapping ratio of all coordinate points. In this embodiment, the size of the matrix ratioGB is 256*256. Here, all coordinate points refer not only to all coordinate points in an image, but also to all coordinate points in all images used to establish a dictionary. The formula for the third mapping ratio of the coordinate point (i, j) in the matrix ratioGB is:

[0122] ratioGB(round(greenLM(i,j)),round(bLM(i,j)))=ratioGB(round(greenLM(i,j)),round(bLM(i,j)))+gLM(i,j) / bLM(i,j)When multiple identical index conditions of the third mapping ratio appear, this formula accumulates the multiple third mapping ratios. Therefore, a matrix ratioGBN is required to count and save the number of occurrences of the index conditions of the third mapping ratio. In this embodiment, the size of the matrix ratioGBN is 256*1. If the same index condition of the second mapping ratio appears, the statistical number is increased by 1. The formula of the matrix ratioGBN is:

[0123] ratioGBN(round(greenLM(i,j)),round(bLM(i,j)))=ratioGBN(round(greenLM(i,j)),round(bLM(i,j)))+1;

[0124] The round function in the formula is the rounding function.

[0125] When multiple index conditions for the same third mapping ratio appear, the average of the multiple third mapping ratios is obtained by dividing the value of the matrix ratioGB by the value of the matrix ratioGBN. In this embodiment, this average is used as the index result in the third dictionary model. Those skilled in the art may also use a statistical value such as the maximum value or median as the index result.

[0126] Example 2

[0127] In step 2-1 of the embodiment 1, if there is no true color image corresponding to the target image, the following steps are performed: Figure 5 As shown:

[0128] Step A: Acquire the three-band image of the sample image in Example 1, and obtain the feature descriptor of each coordinate point in the selected band. The selected band and the selected channel in step 3 may correspond to each other. For example, if the selected channel is the G channel, the selected band is green light imaging. If the selected channel is the R channel, the selected band is red light imaging. If the selected band is the B channel, the selected band is blue light imaging.

[0129] In step B, the feature descriptor of each coordinate point in the selected band is used as an index condition, the corresponding index result is indexed in the fourth dictionary model, and the index result is used as the image feature of the three channels of the coordinate point to obtain the first channel feature, the second channel feature and the third channel feature of the coordinate point.

[0130] Among them, the fourth dictionary model is constructed by the following method:

[0131] Step 1: Acquire a true color image from the sample image. Here, the true color image may not be correlated with the three-band image in Example 1. Correlation here means that the true color image and the three-band image are collected for the same target.

[0132] Step 2: Obtain the image features of each coordinate point in the three channels of the true color image in the sample image, which are the first channel feature, the second channel feature, and the third channel feature. For example, the first channel feature is the B channel feature, the second channel feature is the R channel feature, and the third channel feature is the G channel feature.

[0133] Step 3: Calculate the descriptor of each coordinate point in the selected channel. In this embodiment, the G channel with the highest contrast is set as the selected channel, and the method of establishing the feature descriptor of any coordinate point (i, j) in the G channel image is specifically described:

[0134] Step 3-1: Calculate the centroid coordinates (x, y) of the image block centered at point (i, j) and with length and width m in the G channel image according to the grayscale centroid method. The method for calculating the centroid coordinates (x, y) is a prior art and will not be described in detail in this embodiment.

[0135] Step 3-2: Calculate the vector from point (i, j) to its centroid coordinates (x, y), and use the direction of the vector as the angular direction of point (i, j).

[0136] Step 3-3, calculate the feature descriptor of point (i, j) in the G channel image according to the angle direction and the grayscale information of the image block. This embodiment uses BRIEF feature descriptor for calculation, and SIFT descriptor can also be used, such as Figure 6 As shown, a circle O is constructed with point (i, j) as the center and d as the radius. N point pairs are then selected within circle O using a point pair selection rule, such as random angles and the same distance between the first point in the pair and the center. Each point pair contains two mirror-image points, with the diameter passing through point (i, j) as the mirror axis. The point selected according to the point pair selection rule is the first point, and the mirror point corresponding to the first point is the second point, i.e., the point pair contains both the first and second points. The pixel values ​​(i.e., grayscale values) of the two points in each point pair are then compared. If the first point is greater than the second point, the value is 1; otherwise, the value is 0. The comparison values ​​of all point pairs are combined to obtain a set of feature values ​​composed of 1s and 0s. These feature values ​​are the feature descriptors for point (i, j) in the G channel image. In this embodiment, the feature descriptor is a 64-bit binary code, which is converted into eight decimal digits (0 to 255) corresponding to 8-bit binary codes.

[0137] Step 4: Use the first channel features, second channel features, and third channel features as index results, establish an association relationship between the descriptor and the index results, and count the association relationships for each coordinate point to obtain a fourth dictionary model. Specifically, use the first channel features, second channel features, and third channel features of the coordinate point (i, j) as the index results, associate the feature descriptor of the coordinate point (i, j) in the G channel with the index results, store the association relationship between the feature descriptor and the index results, and build a feature descriptor index library.

[0138] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. An image fusion method based on a dictionary model, characterized in that: The following steps are involved: Step 1: Obtain the image features of each coordinate point of the image to be fused in the three band images, namely the first band feature, the second band feature and the third band feature; Step 2: For each coordinate point, obtain a first mapping ratio by indexing in the first dictionary model based on the first band feature, and calculate a first channel index value according to the first band feature and the first mapping ratio; Obtain the first channel feature according to the first channel index value; Obtain a second mapping ratio by indexing the second dictionary model based on the second band feature and the first channel feature, and obtain a second channel index value by calculating according to the second band feature and the second mapping ratio; Obtain a third mapping ratio by indexing in a third dictionary model based on the third band feature and the first channel feature, and obtain a third channel index value by calculating according to the third band feature and the third mapping ratio; The method for constructing the first dictionary model, the second dictionary model, and the third dictionary model includes the following steps: Step 2-1, obtaining the image features of each coordinate point of the sample image in the three band images, namely the first band feature, the second band feature and the third band feature; Obtain the image features of the three channels of each coordinate point, namely the first channel feature, the second channel feature and the third channel feature; Step 2-2: For a single coordinate point, calculate a first mapping ratio between the first band feature and the first channel feature, use the first band feature as an index condition, establish an association relationship between the index condition and the first mapping ratio, and count the association relationships for each coordinate point to obtain a first dictionary model. For a single coordinate point, a second mapping ratio of the second channel feature to the first channel feature is calculated, the second band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the second mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a second dictionary model; For a single coordinate point, a third mapping ratio of the third channel feature to the first channel feature is calculated, the third band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the third mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a third dictionary model; Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image.

2. The image fusion method based on a dictionary model according to claim 1, characterized in that: For a single coordinate point, a local area around the coordinate point is selected based on a preset size, and the image feature is a statistical value of the grayscale of the local area.

3. The image fusion method based on a dictionary model according to claim 1, characterized in that: In step 2, if there is a value greater than 255 in the first channel index value of all coordinate points, normalization processing is performed and then multiplied by 255; Alternatively, if the second channel index values ​​of all coordinate points have a value greater than 255, normalization is performed and then multiplied by 255; Alternatively, if the third channel index values ​​of all coordinate points have a value greater than 255, normalization processing is performed and then multiplied by 255.

4. The image fusion method based on a dictionary model according to claim 1, characterized in that: Before step 2-1, the method further includes obtaining a band image and a true color image collected for the same target as a sample image; The image features of the three channels of each coordinate point obtained in step 2-1 are the image features of each coordinate point in the three channels of the true color image, which are the first channel features, the second channel features and the third channel features.

5. The image fusion method based on a dictionary model according to claim 1, characterized in that: In step 2-2, the statistical association relationship of each coordinate point to obtain the first dictionary model specifically comprises: when the index conditions of the first mapping ratios of multiple coordinate points are the same, taking the average of the multiple first mapping ratios as the index result of the index condition; And / or, the statistical association relationship of each coordinate point to obtain the second dictionary model specifically comprises: when the index conditions of the second mapping ratios of multiple coordinate points are the same, taking the average of the multiple second mapping ratios as the index result of the index condition; And / or, the statistical association relationship of each coordinate point to obtain the third dictionary model is specifically: when the index conditions of the third mapping ratios of multiple coordinate points are the same, the average of the multiple third mapping ratios is used as the index result of the index condition.

6. The image fusion method based on a dictionary model according to claim 1, characterized in that: If more than three bands of images are obtained in step 2-1, the band images with similar bands are merged, and finally three images are retained, and image features are extracted from each of them.

7. The image fusion method based on a dictionary model according to claim 1, characterized in that: The step 2-1 of obtaining the image features of the three channels of each coordinate point specifically includes: Calculate the descriptor of each coordinate point in the selected band image; For a single coordinate point, the index results of the first channel feature, the second channel feature and the third channel feature are obtained by indexing in the fourth dictionary model based on the descriptor, and the index results are used as the image features of the three channels of the coordinate point.

8. The image fusion method based on a dictionary model according to claim 7, characterized in that: Obtain image features in three channels of each coordinate point of the true color image in the sample image, namely, first channel features, second channel features, and third channel features, and calculate the descriptor of each coordinate point in the selected channel; The first channel feature, the second channel feature and the third channel feature are used as index results, an association relationship between the descriptor and the index result is established, and the association relationship of each coordinate point is counted to obtain a fourth dictionary model.

9. The image fusion method based on a dictionary model according to claim 8, characterized in that: The method of calculating the descriptor of each coordinate point includes: Step 3-1, calculate the centroid coordinates (x, y) of the image block centered at coordinate point (i, j) and with a side length of m in the image using the grayscale centroid method; Step 3-2, calculate the vector from the coordinate point (i, j) to the centroid coordinate (x, y), and the direction of the vector is used as the angular direction of the coordinate point (i, j); Step 3-3, calculating the descriptor of point (i, j) according to the angle direction and the grayscale information of the image block.

Images