Image fusion method based on dictionary model

Abstract

The invention discloses an image fusion method based on a dictionary model, and the method comprises the steps: obtaining the image features of each coordinate point of a to-be-fused image in three wave band images, namely, a first wave band feature, a second wave band feature and a third wave band feature; for each coordinate point, indexing and calculating in a first dictionary model based on the first wave band feature to obtain a first channel index value; indexing in a second dictionary model based on the second wave band feature and the first channel feature, and calculating to obtain a second channel index value; indexing in a third dictionary model based on the third wave band feature and the first channel feature, and calculating to obtain a third channel index value; and carrying out superposition processing on the first channel index value, the first channel index value and the third channel index value of each coordinate point to obtain a fused true color image. According to the method, the mapping proportion relation of three-channel and three-band image features is established, and the true color image is rapidly restored in a hierarchical feature index mode.

Description

Technical Field

[0001] The invention relates to the technical field of digital image processing, and in particular to an image fusion method based on a dictionary model. Background Art

[0002] Medical imaging technology has developed rapidly and has become an indispensable technology in medical diagnosis. Since the advent of the digital imaging era, the generation of massive data has provided more possibilities for the future development of medical imaging. Therefore, how to further analyze and mine medical imaging big data, how to extract valuable information from high-dimensional data of medical images, and how to closely integrate the development of modern medical imaging with precision medicine have become important topics for the future development of medical imaging.

[0003] Currently, due to the different degrees of update and iteration of different medical devices, some medical devices can output true color images, while some medical devices can only output images of three or more bands. The color saturation of true color images can well reflect the subtle changes in the structure of human tissues and is closer to the subjective color perception of the human eye. If the images of the three bands are simply superimposed, the superimposed image effect is dim and the color saturation is low. The problem is very obvious, which is not conducive to medical staff to observe the lesion or plan surgery. The existing technology also uses a deep learning method to fuse the images of the three bands, but this method is relatively complex, the processing speed is slow, and the requirements for the equipment are high. Therefore, a technology that can quickly fuse the three-band images to generate true color images is needed. Summary of the invention

[0004] To this end, an embodiment of the present invention provides an image fusion method based on a dictionary model to solve the problems in the prior art of poor image effect and low color saturation due to simple superposition of three-band images and low efficiency of existing true color image fusion methods.

[0005] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0006] An image fusion method based on a dictionary model comprises the following steps:

[0007] Step 1: Obtain the image features of each coordinate point of the image to be fused in the three band images, which are the first band feature, the second band feature and the third band feature.

[0008] Step 2: For each coordinate point, the first mapping ratio is obtained by indexing in the first dictionary model based on the first band feature, and the first channel index value is calculated based on the first band feature and the first mapping ratio. The first channel feature is obtained based on the first channel index value, and the second mapping ratio is obtained by indexing in the second dictionary model based on the second band feature and the first channel feature, and the second channel index value is calculated based on the second band feature and the second mapping ratio; the third mapping ratio is obtained by indexing in the third dictionary model based on the third band feature and the first channel feature, and the third channel index value is calculated based on the third band feature and the third mapping ratio.

[0009] Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image. 。

[0010] Furthermore, for a single coordinate point, a local area around the coordinate point is selected based on a preset size, and the image feature is a statistical value of the grayscale of the local area.

[0011] Furthermore, in step 2, if there is a value greater than 255 in the first channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255; or, if there is a value greater than 255 in the second channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255; or, if there is a value greater than 255 in the third channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255.

[0012] Furthermore, the method for constructing the first dictionary model, the second dictionary model and the third dictionary model in step 2 includes the following steps:

[0013] Step 2-1, obtaining image features of each coordinate point of the sample image in three band images, which are first band features, second band features and third band features; obtaining image features of three channels of each coordinate point, which are first channel features, second channel features and third channel features;

[0014] Step 2-2, for a single coordinate point, calculate the first mapping ratio of the first band feature to the first channel feature, use the first band feature as an index condition, establish an association relationship between the index condition and the first mapping ratio, and count the association relationship of each coordinate point to obtain a first dictionary model. For a single coordinate point, calculate the second mapping ratio of the second channel feature to the first channel feature, use the second band feature and the first channel feature as index conditions, establish an association relationship between the index condition and the second mapping ratio, and count the association relationship of each coordinate point to obtain a second dictionary model. For a single coordinate point, calculate the third mapping ratio of the third channel feature to the first channel feature, use the third band feature and the first channel feature as index conditions, establish an association relationship between the index condition and the third mapping ratio, and count the association relationship of each coordinate point to obtain a third dictionary model.

[0015] Furthermore, before step 2-1, it also includes obtaining a band image and a true color image collected for the same target as a sample image; the image features of the three channels of each coordinate point obtained in step 2-1 are the image features of each coordinate point in the three channels of the true color image, which are the first channel features, the second channel features and the third channel features respectively.

[0016] Furthermore, in step 2-2, the statistical association of each coordinate point to obtain the first dictionary model is specifically: when the index conditions of the first mapping ratios of multiple coordinate points are the same, the average of the multiple first mapping ratios is used as the index result of the index condition. And / or, the statistical association of each coordinate point to obtain the second dictionary model is specifically: when the index conditions of the second mapping ratios of multiple coordinate points are the same, the average of the multiple second mapping ratios is used as the index result of the index condition. And / or, the statistical association of each coordinate point to obtain the third dictionary model is specifically: when the index conditions of the third mapping ratios of multiple coordinate points are the same, the average of the multiple third mapping ratios is used as the index result of the index condition.

[0017] Furthermore, if images of more than three bands are obtained in step 2-1, band images with similar bands are merged, and finally three images are retained, and image features are extracted from each of them.

[0018] Furthermore, the image features of the three channels of each coordinate point are obtained in the step 2-1, specifically including: calculating the descriptor of each coordinate point in the selected band image; for a single coordinate point, obtaining the index results of the first channel feature, the second channel feature and the third channel feature based on the index of the descriptor in the fourth dictionary model, and using the index results as the image features of the three channels of the coordinate point.

[0019] Furthermore, image features in three channels of each coordinate point of the true color image in the sample image are obtained, which are the first channel features, the second channel features and the third channel features, and the descriptor of each coordinate point in the selected channel is calculated; the first channel features, the second channel features and the third channel features are used as index results, an association relationship between the descriptor and the index result is established, and the association relationship of each coordinate point is counted to obtain a fourth dictionary model.

[0020] Furthermore, the method for calculating the descriptor of each coordinate point includes:

[0021] Step 3-1, calculate the centroid coordinates (x, y) of the image block centered at the coordinate point (i, j) and with a side length of m in the image according to the grayscale centroid method.

[0022] Step 3-2, calculate the vector from the coordinate point (i, j) to the centroid coordinate (x, y), and the direction of the vector is used as the angular direction of the coordinate point (i, j).

[0023] Step 3-3, calculating the descriptor of point (i, j) according to the angle direction and the grayscale information of the image block.

[0024] The embodiments of the present invention have the following advantages:

[0025] The present invention establishes a mapping ratio relationship between the image features of three channels and three bands. Compared with the method of simply superimposing three-band images, the color of the image after fusion by this method will be close to the image sample, with higher color richness and closer to the color of the subjective perception of the human eye. In addition, this method can be used in medical images to more accurately observe the lesions, effectively reduce the misdiagnosis rate, and be used in medical devices to facilitate surgical planning and real-time monitoring during surgery, improve surgical accuracy, reliability and safety, and have high application value. In addition, compared with the fusion method using deep learning, the present invention has a simpler processing flow, lower requirements on equipment, can be widely used in various types of medical imaging equipment, and has a faster speed, which can meet the requirements of real-time output of color images in some special scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0027] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in mapping proportions, or adjustment in size should still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0028] Figure 1 A method flow chart of an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0029] Figure 2 A method flow chart of a dictionary use phase in a dictionary model-based image fusion method provided in Embodiment 1 of the present invention;

[0030] Figure 3 A comparison diagram of a simple superposition image of three-band images of an image to be fused and a restored true color image in an image fusion method based on a dictionary model provided in Example 1 of the present invention;

[0031] Figure 4 A method flow chart of a dictionary construction phase in a dictionary model-based image fusion method provided in Embodiment 1 of the present invention;

[0032] Figure 5 A method flow chart of a dictionary use phase in a dictionary model-based image fusion method provided in Embodiment 2 of the present invention;

[0033] Figure 6 for Figure 5 Flowchart of the method for calculating feature descriptors in . DETAILED DESCRIPTION

[0034] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Example 1

[0036] An image fusion method based on dictionary model, such as Figure 1 As shown, the following steps are included:

[0037] Step 1, respectively obtain the image features of each coordinate point in the three band images of the image to be fused, which are the first band features, the second band features and the third band features.

[0038] Combined with the previous example, the type of the three-band image to be fused and the three-band image used to construct the dictionary model can be the same or different. In this embodiment, the same type is selected, and any one of the three-band images is used as the first band image, and the remaining two images are respectively the second band image and the third band image. In this embodiment, the blue light imaging image in the three-band image is set as the first band image, the red light imaging image is set as the second band image, and the green light imaging image is set as the third band image. If more than three bands of images are obtained, the images with similar bands are merged, and finally three images are retained, and image features are extracted respectively. The preferred three bands are preferably close to the red light, green light, and blue light bands, respectively. The merging method is such as taking the grayscale mean or weighted average or maximum value of the close band images. For a single coordinate point, a local area around the coordinate point is selected based on a preset size, and the image feature is the statistical value of the grayscale of the local area. The statistical value can be the mean, maximum value, median, etc. In this embodiment, the first band feature of the coordinate point (i, j) is the grayscale value mean blueLM(i, j) of the local area around the coordinate point (i, j) in the blue light imaging image of the image to be fused, the second band feature is the grayscale value mean redLM(i, j) of the local area around the coordinate point (i, j) in the red light imaging image of the image to be fused, and the third band feature is the grayscale value mean greenLM(i, j) of the local area around the coordinate point (i, j) in the green light imaging image of the image to be fused.

[0039] The first band feature, the second band feature and the third band feature may be image features of the same nature or image features of different natures. As long as the technical solutions that can be used to reflect the image features at the coordinate points in the first band image, the second band image and the third band image fall within the protection scope of the present invention. The image feature may be the image feature of a single coordinate point or the image feature of a local area composed of multiple coordinate points:

[0040] The image feature of a single coordinate point may be a grayscale value (ie, pixel value), a contrast value, etc. of a single pixel.

[0041] The local area composed of multiple coordinate points can be set to a variety of shapes, such as rectangle, square, circle, etc. The size parameters of the local area are determined by its shape. For example, if the local area is a rectangle, the size parameters are length and width, and the selected coordinate points can be special points such as the intersection of the diagonals of the rectangle, corner points, and center points of the edges; if the local area is a circle, the size parameter is the radius, and the selected coordinate point is the center of the circle. The image features of the local area include one or more of the grayscale value extreme value, grayscale value mean value, contrast value extreme value, contrast value mean value, gradient value extreme value, gradient value mean value, histogram mean value, local grayscale variance, local / global grayscale variance mean value, Brenner gradient, Laplace gradient, Sobel gradient, Tenegrad gradient, SMD gradient, SMD2 gradient, Roberts gradient, energy gradient, local / global information entropy, grayscale co-occurrence matrix feature, Tamura texture feature, autoregressive texture model, wavelet transform feature, energy spectrum function feature, and LBP texture feature of all pixels in the local area, where the extreme value mentioned is the maximum value, the minimum value, the sum value, etc. In this embodiment, it is impossible to enumerate the features of a single pixel or multiple pixels. As long as the features can be used as a single coordinate point or multiple coordinate points, they fall within the protection scope of the present invention.

[0042] Taking the gray value mean of the local area of ​​the point (i, j) in the first band image as the first band feature of the point (i, j) as an example, the local area is a square with a side length of n and the point (i, j) as the center, and the first band feature is the gray value mean blueLM(i, j) of the local area of ​​the point (i, j) in the first band image, and the formula is as follows:

[0043]

[0044] Where:

[0045] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0046] imageBLUE(p,q) is the grayscale value of the pixel at the coordinate point (p,q) in the first band image;

[0047] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0048] Taking the gray value mean of the local area of ​​the point (i, j) in the second band image as the second band feature as an example, the local area is a square with a side length of n and a point (i, j) as the center, and the second band feature is the gray value mean redLM(i, j) of the local area of ​​the point (i, j) in the second band image, and the formula is as follows:

[0049]

[0050] Where:

[0051] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0052] imageRED(p,q) is the grayscale value of the pixel at the coordinate point (p,q) in the second band image;

[0053] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0054] Taking the gray value mean of the local area of ​​the point (i, j) in the third band image as the third band feature as an example, the local area is a square centered at point (i, j) and with a side length of n. The third band feature is the gray value mean greenLM(i, j) of the local area of ​​the point (i, j) in the third band image. The formula is as follows:

[0055]

[0056] Where:

[0057] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0058] imageGREEN(p,q) is the gray value of the pixel at the coordinate point (p,q) in the third band image;

[0059] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0060] Step 2, for each coordinate point, obtain the first mapping ratio by indexing in the first dictionary model based on the first band feature, and calculate the first channel index value according to the first band feature and the first mapping ratio; obtain the first channel feature according to the first channel index value; obtain the second mapping ratio by indexing in the second dictionary model based on the second band feature and the first channel feature, and calculate the second channel index value according to the second band feature and the second mapping ratio; obtain the third mapping ratio by indexing in the third dictionary model based on the third band feature and the first channel feature, and calculate the third channel index value according to the third band feature and the third mapping ratio.

[0061] like Figure 2 As shown, the following takes the coordinate point (i, j) as an example to illustrate the specific method of obtaining the first channel index value, the second channel index value and the third channel index value of the coordinate point (i, j):

[0062] a. Index value of the first channel

[0063] Get the first band feature blueLM(i,j) of the coordinate point (i,j) in the first band image of the image to be fused, use the first band feature blueLM(i,j) as the index condition to index in the pre-built first dictionary model, and get its corresponding first mapping ratio blueLM(i,j) / bLM(i,j) , and then calculate the first channel index value blueAfter(i,j) of the coordinate point (i,j) according to the first band feature blueLM(i,j) and the first mapping ratio. The calculation method of the first channel index value varies according to the different methods for generating the first mapping ratio. In this embodiment, the calculation method of the first channel index value is to calculate the first band feature blueLM(i,j) and the first mapping ratio blueLM(i,j) / bLM(i,j) The quotient of , we get the first channel index value blueAfter(i,j) of the coordinate point (i,j). The specific formula is as follows:

[0064] blueAfter(i,j)=blueLM(i,j) / ratioB(round(blueLM(i,j)),1));

[0065] In the formula, the round function is a rounding function, ratioB(round(blueLM(i,j)),1)) is the value in the matrix that stores the association relationship of the first mapping ratio. If ratioB(round(blueLM(i,j)),1)) or round(blueLM(i,j)) is 0, the formula is not executed; if there is a value greater than 255 in the first channel features of all fused coordinate points, normalization is performed and then multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent the existence of erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all fused coordinate points are not greater than 255, they can still be normalized and multiplied by 255, which can uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0066] The first channel feature is obtained according to the first channel index value. Specifically, after obtaining the first channel index values ​​blueAfter of all the coordinate points of the image to be fused, for each coordinate point, the first channel index value blueAfter(i, j) is used as the grayscale value of the coordinate point (i, j) in the first channel image obtained by the index, and the first channel feature blueAfterLM(i, j) of the coordinate point is obtained by referring to the calculation method of the first band feature of the aforementioned coordinate point, that is, the first channel feature of the point (i, j) is the grayscale value mean blueAfterLM(i, j) of the square local area with the coordinate point (i, j) as the midpoint and the side length n in the first channel image obtained by the index.

[0067] b. Second channel index value

[0068] The second band feature redLM(i,j) of the coordinate point (i,j) in the second band image of the image to be fused is obtained, that is, the grayscale mean of the local area formed by the square with the coordinate point (i,j) as the center and the side length n is used as the second band feature redLM(i,j) of the coordinate point (i,j). The second band feature redLM(i,j) and the first channel feature blueAfterLM(i,j) obtained by the index calculation are used as index conditions to index in the second dictionary model to obtain the second mapping ratio rLM(i,j) / bLM(i,j), and then the second channel index value redAfter(i,j) of the coordinate point (i,j) is calculated based on the first channel feature blueAfterLM(i,j) of the coordinate point (i,j) and the second mapping ratio. The calculation method of the second channel index value varies according to the method for generating the second mapping ratio. In this embodiment, the calculation method of the second channel index value is to calculate the product of the first channel feature blueAfterLM(i,j) and the second mapping ratio rLM(i,j) / bLM(i,j) to obtain the second channel index value redAfter(i,j) of the coordinate point (i,j). The specific formula is as follows:

[0069] redAfter(i,j)=ratioRB(round(redLM(i,j)),round(blueAfterLM(i,j)))*blueAfterLM(i,j);

[0070] In the formula, round function is a rounding function, ratioRB(round(redLM(i,j)),round(blueAfterLM(i,j))) is the value in the matrix that stores the association relationship of the second mapping ratio. If any one of ratioRB(round(redLM(i,j)), round(blueAfterLM(i,j))), round(redLM(i,j)), round(blueAfterLM(i,j)), blueAfterLM(i,j) is 0, the formula is not executed; if there is a value greater than 255 in the second channel features of all fused coordinate points, normalization processing is performed and then multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent the existence of erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all fused coordinate points are not greater than 255, normalization processing can still be performed and then multiplied by 255, which can uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0071] c. The third channel index value

[0072] The third band feature greenLM(i,j) of the coordinate point (i,j) in the third band image of the image to be fused is obtained, that is, the grayscale mean of the local area formed by the square with the coordinate point (i,j) as the center and the side length n is used as the third band feature greenLM(i,j) of the coordinate point (i,j). The third band feature greenLM(i,j) and the first channel feature blueAfterLM(i,j) obtained by the index calculation are used as index conditions to index in the third dictionary model to obtain the third mapping ratio gLM(i,j) / bLM(i,j), and then the third channel index value greenAfter(i,j) of the coordinate point (i,j) is calculated based on the first channel feature blueAfterLM(i,j) of the coordinate point (i,j) and the third mapping ratio. The calculation method of the third channel feature varies according to the generation method of the third mapping ratio. In this embodiment, the calculation method of the third channel feature is to calculate the product of the first channel feature blueAfterLM(i,j) and the third mapping ratio gLM(i,j) / bLM(i,j) to obtain the third channel index value greenAfter(i,j) of the coordinate point (i,j). The specific formula is as follows:

[0073] greenAfter(i,j)=ratioGB(round(greenLM(i,j)),round(blueAfterLM(i,j)))*blueA afterLM(i,j);

[0074] In the formula, round function is a rounding function, ratioGB(round(greenLM(i,j)), round(blueAfterLM(i,j))) is a value in a matrix that stores the association relationship of the third mapping ratio. If any one of ratioGB(round(greenLM(i,j)), round(blueAfterLM(i,j))), round(greenLM(i,j)), round(blueAfterLM(i,j)), blueAfterLM(i,j) is 0, the formula is not executed; if there is a value greater than 255 in the third channel index value of all fused coordinate points, normalization processing is performed and then multiplied by 255. This operation can adjust the grayscale value of the image as a whole to prevent the existence of erroneous brightness values ​​exceeding 255. In addition, even if the first channel features of all fused coordinate points are not greater than 255, normalization processing can still be performed and then multiplied by 255, which can uniformly adjust the grayscale value of the image to prevent the overall image from being too bright or too dark.

[0075] Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image. 。 like Figure 3 The image on the left is the image obtained by simply superimposing the three-band images to be fused. Figure 3 The image on the right is the true color image after the three-band image is fused by this method. Figure 3 From the left and right images in the middle, we can intuitively see that the color saturation of the transformed true color image is richer, and the color is more consistent with the color observed by the human eye, the image presentation is more realistic, and the image quality is significantly improved. In addition, this method can be used in medical imaging to more accurately observe lesions and effectively reduce the misdiagnosis rate. It is also used in medical devices to facilitate surgical planning and real-time monitoring during surgery, improving surgical accuracy, reliability, and safety.

[0076] The following is a detailed description of the construction method of the first dictionary model, the second dictionary model and the third dictionary model in step 2 with examples:

[0077] Step 2-1, obtaining the image features of each coordinate point of the sample image in the three band images, which are the first band features, the second band features and the third band features, and obtaining the image features of the three channels of each coordinate point, which are the first channel features, the second channel features and the third channel features.

[0078] Before step 2-1, the method further includes acquiring three band images and a true color image for the same target as sample images, and then acquiring the image features of each coordinate point in the three band images to obtain the first band feature, the second band feature and the third band feature of each coordinate point. And acquiring the image features of each coordinate point in the three channels of the true color image, which are the first channel feature, the second channel feature and the third channel feature. Specifically, it includes:

[0079] Step 2-1-1, respectively obtain the three-band images of the current image to obtain the first band image, the second band image and the third band image.

[0080] In this embodiment, the three-band image includes a red light imaging image, a blue light imaging image, and a green light imaging image. In this technology, any one of the three-band images is used as the first band image, and the other two images are respectively the second band image and the third band image. In this embodiment, the blue light imaging image is set as the first band image, the red light imaging image is set as the second band image, and the green light imaging image is set as the third band image. It should be emphasized that if the image features of band images with more than three bands are obtained, the image features of the bands close to each other are merged so that the images with more than three bands are merged into images of three bands. The merging method may be, for example, taking the average of the close bands.

[0081] Step 2-1-2, respectively obtain the first band features of each coordinate point in the current image in the first band image, the second band features in the second band image, and the third band features in the third band image. Combined with the previous example, the first band features are the image features of each coordinate point in the blue light imaging image, the second band features are the image features of each coordinate point in the red light imaging image, and the third band features are the image features of each coordinate point in the green light imaging image.

[0082] Step 2-1-3, for the true color images collected for the same target, three-channel images of the true color images are obtained respectively to obtain a first channel image, a second channel image and a third channel image. In this embodiment, they are specifically a B channel image, an R channel image and a G channel image. In this embodiment, the B channel image is set as the first channel image, the R channel image is set as the second channel image, and the G channel image is set as the third channel image.

[0083] Step 2-1-4, respectively obtain the first channel feature of each coordinate point located in the first channel image, the second channel feature located in the second channel image, and the third channel feature located in the third channel image. In order to facilitate subsequent calculations, the first channel feature is an image feature of the same type as the first band feature, the second channel feature is an image feature of the same type as the second band feature, and the third channel feature is an image feature of the same type as the third band feature.

[0084] Combined with the previous example, the first band feature of point (i, j) is the gray value mean blueLM(i, j) of the local square area with the coordinate point (i, j) as the midpoint and the side length n in the first band image. The first channel feature of point (i, j) is the gray value mean bLM(i, j) of the local square area with the coordinate point (i, j) as the midpoint and the side length n in the first channel image. The formula is as follows:

[0085]

[0086] Where:

[0087] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0088] imageB(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0089] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0090] Combined with the previous example, the second band feature is the gray value mean redLM(i,j) of the local area of ​​the square with the coordinate point (i,j) as the midpoint and the side length n in the second band image, and the second channel feature is the gray value mean rLM(i,j) of the local area of ​​the coordinate point (i,j) in the second channel image. The formula is as follows:

[0091]

[0092] Where:

[0093] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0094] imageR(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0095] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0096] Combined with the previous example, the third band feature is the gray value mean greenLM(i,j) of the local area of ​​the square with the coordinate point (i,j) as the midpoint and the side length n in the third band image, and the third channel feature is the gray value mean gLM(i,j) of the local area of ​​the coordinate point (i,j) in the third channel image. The formula is as follows:

[0097]

[0098] Where:

[0099] (p,q) is the pixel coordinate in the local area, p is the horizontal coordinate and q is the vertical coordinate;

[0100] imageG(p,q) is the pixel grayscale value of the coordinate point (p,q) in the first channel image;

[0101] n is the size parameter of the local area, that is, the number of pixels contained in the side length of the square.

[0102] Step 2-2, for a single coordinate point, calculate the first mapping ratio of the first band feature to the first channel feature, establish an association relationship between the first band feature and the first mapping ratio, count the association relationship of each coordinate point, and obtain a first dictionary model.

[0103] For a single coordinate point, a second mapping ratio of the second channel feature to the first channel feature is calculated, the second band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the second mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a second dictionary model;

[0104] For a single coordinate point, the third mapping ratio of the third channel feature to the first channel feature is calculated, the third band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the third mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a third dictionary model.

[0105] like Figure 4 As shown, the generation method of the first dictionary model, the second dictionary model, and the third dictionary model is specifically described below in combination with the previous example:

[0106] 2-2-1, the first dictionary model

[0107] For a single coordinate point, calculate the first mapping ratio of the first band feature and the first channel feature, set the first band feature as the index condition of the first mapping ratio, and establish the association relationship between the first band feature and the first mapping ratio. Taking the coordinate point (i, j) as an example, calculate the first mapping ratio of the first band feature blueLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j), that is, blueLM(i,j) / bLM(i,j) , and vice versa. The first band feature blueLM(i,j) is set as the index condition of the first mapping ratio of the coordinate point (i,j), and the association relationship between the first band feature and the first mapping ratio is established. A matrix ratioB can be set to save the association relationship of the first mapping ratio of all coordinate points. Here, all coordinate points not only refer to all coordinate points in an image, but also all coordinate points in all images used to establish the dictionary. In this embodiment, the size of the matrix ratioB is 256*1, and the formula for the first mapping ratio of the coordinate point (i,j) in the matrix ratioB is:

[0108] ratioB(round(blueLM(i,j)),1)=ratioB(round(blueLM(i,j)),1)+blueLM(i,j) / bLM(i,j);

[0109] When multiple index conditions of the same first mapping ratio appear, the formula accumulates the multiple first mapping ratios. Therefore, a matrix ratioBN is also required to count and save the number of occurrences of the first band feature. In this embodiment, the size of the matrix ratioBN is 256*1. If the same first band feature appears, the statistical number is increased by 1. The formula of the matrix ratioBN is:

[0110] ratioBN((round(blueLM(i,j)),1)=ratioBN((round(blueLM(i,j)),1)+1;

[0111] The round function in the formula is a rounding function.

[0112] When multiple identical first band features appear, the value of the matrix ratioB is divided by the value of the matrix ratioBN to obtain the mean of the first mapping ratios corresponding to the first band features. In this embodiment, the mean is used as the index result in the first dictionary model. Those skilled in the art may also use statistical values ​​such as the maximum value or the median as the index result.

[0113] 2-2-2, the second dictionary model

[0114] For a single coordinate point, the second mapping ratio of the second channel feature to the first channel feature is calculated, and the second band feature and the first channel feature are used as index conditions to establish an association relationship between the index condition and the second mapping ratio. Taking the coordinate point (i, j) as an example, the second mapping ratio of the second channel feature rLM(i, j) of the coordinate point (i, j) and the first channel feature bLM(i, j) is calculated, that is, rLM(i, j) / bLM(i, j), and vice versa; the combination of the second band feature redLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j) is set as the index condition of the second mapping ratio, and the association relationship between the index condition of the second mapping ratio and the second mapping ratio is established. A matrix ratioRB can be set to save the association relationship of the second mapping ratios of all coordinate points, where all coordinate points not only refer to all coordinate points in an image, but also all coordinate points in all images used to establish a dictionary. In this embodiment, the size of the matrix ratioRB is 256*256, and the formula for the second mapping ratio of the coordinate point (i, j) in the matrix ratioRB is:

[0115] ratioRB(round(redLM(i,j)),round(bLM(i,j)))=ratioRB(round(redLM(i,j)),round(bLM(i,j)))+rLM(i,j) / bLM(i,j)

[0116] When multiple identical index conditions of the second mapping ratio appear, the formula accumulates the multiple second mapping ratios. Therefore, a matrix ratioBN is required to count and save the number of occurrences of the index conditions of the second mapping ratio. In this embodiment, the size of the matrix ratioBN is 256*1. If the identical index condition of the second mapping ratio appears, the statistical number is increased by 1. The formula of the matrix ratioBN is:

[0117] ratioRBN(round(redLM(i,j)),round(bLM(i,j)))=ratioRBN(round(redLM(i,j)),round(bLM(i,j)))+1;

[0118] The round function in the formula is a rounding function.

[0119] When multiple index conditions of the same second mapping ratio appear, the value of the matrix ratioRB is divided by the value of the matrix ratioRBN to obtain the mean of the multiple second mapping ratios. In this embodiment, the mean is used as the index result in the second dictionary model. Those skilled in the art may also use statistical values ​​such as the maximum value or the median as the index result.

[0120] 2-2-3, the third dictionary model

[0121] For a single coordinate point, the third mapping ratio of the third channel feature to the first channel feature is calculated, and the third band feature and the first channel feature are used as index conditions to establish an association relationship between the index condition and the third mapping ratio. Taking the coordinate point (i, j) as an example, the third mapping ratio of the third channel feature gLM(i, j) of the coordinate point (i, j) and the first channel feature bLM(i, j) is calculated, that is, gLM(i, j) / bLM(i, j), and vice versa; the combination of the third band feature greenLM(i, j) and the first channel feature bLM(i, j) of the coordinate point (i, j) is set as the index condition of the third mapping ratio, and an association relationship between the index condition of the third mapping ratio and the third mapping ratio is established. A matrix ratioGB can be set to save the third mapping ratio of all coordinate points. In this embodiment, the size of the matrix ratioGB is 256*256. Here, all coordinate points not only refer to all coordinate points in an image, but also all coordinate points in all images used to establish a dictionary. The formula for the third mapping ratio of the coordinate point (i, j) in the matrix ratioGB is:

[0122] ratioGB(round(greenLM(i,j)),round(bLM(i,j)))=ratioGB(round(greenLM(i,j)),round(bLM(i,j)))+gLM(i,j) / bLM(i,j)When multiple identical index conditions of the third mapping ratio appear, the formula accumulates the multiple third mapping ratios. Therefore, a matrix ratioGBN is required to count and save the number of occurrences of the index conditions of the third mapping ratio. In this embodiment, the size of the matrix ratioGBN is 256*1. If the identical index condition of the second mapping ratio appears, the statistical number is increased by 1. The formula of the matrix ratioGBN is:

[0123] ratioGBN(round(greenLM(i,j)),round(bLM(i,j)))=ratioGBN(round(greenLM(i,j)),round(bLM(i,j)))+1;

[0124] The round function in the formula is a rounding function.

[0125] When multiple index conditions of the same third mapping ratio appear, the value of the matrix ratioGB is divided by the value of the matrix ratioGBN to obtain the mean of the multiple third mapping ratios. In this embodiment, the mean is used as the index result in the third dictionary model. Those skilled in the art may also use statistical values ​​such as the maximum value or the median as the index result.

[0126] Example 2

[0127] In step 2-1 of the embodiment 1, if there is no true color image corresponding to the target image, the following steps are performed: Figure 5 As shown:

[0128] Step A, obtain the three-band image in the sample image in Example 1, and obtain the feature descriptor of each coordinate point in the selected band. The selected band and the selected channel in step 3 may be in a corresponding relationship. For example, if the selected channel is the G channel, the selected band is green light imaging, and if the selected channel is the R channel, the selected band is red light imaging, and if the selected band is the B channel, the selected band is blue light imaging.

[0129] Step B, using the feature descriptors of each coordinate point in the selected band as index conditions, indexing the corresponding index results in the fourth dictionary model, using the index results as the image features of the three channels of the coordinate point, and obtaining the first channel features, second channel features and third channel features of the coordinate point.

[0130] Among them, the fourth dictionary model is constructed by the following method:

[0131] Step 1, obtaining a true color image from the sample image, where the true color image may not be correlated with the three-band image in Example 1. The correlation here means that the true color image and the three-band image are collected for the same target.

[0132] Step 2: Obtain the image features of each coordinate point in the three channels of the true color image in the sample image, which are the first channel feature, the second channel feature and the third channel feature. For example, the first channel feature is the B channel feature, the second channel feature is the R channel feature, and the third channel feature is the G channel feature.

[0133] Step 3: Calculate the descriptor of each coordinate point in the selected channel. In this embodiment, the G channel with the highest contrast is set as the selected channel, and the method of establishing the feature descriptor of any coordinate point (i, j) in the G channel image is specifically described:

[0134] Step 3-1, calculate the centroid coordinates (x, y) of the image block centered at point (i, j) and with length and width m in the G channel image according to the grayscale centroid method. The method for calculating the centroid coordinates (x, y) is a prior art and will not be described in detail in this embodiment.

[0135] Step 3-2: Calculate the vector from point (i, j) to its centroid coordinates (x, y), and use the direction of the vector as the angular direction of point (i, j).

[0136] Step 3-3, calculate the feature descriptor of point (i, j) in the G channel image according to the angle direction and the grayscale information of the image block. This embodiment uses the BRIEF feature descriptor for calculation, and can also use descriptors such as SIFT, such as Figure 6 As shown, a circle O is made with point (i, j) as the center and d as the radius; then N point pairs are selected in circle O according to the point pair selection rule, such as random angles and the same distance between the first point in the point pair and the center of the circle, etc. Each point pair contains two points that are mirror images of each other, and the diameter passing through point (i, j) is used as the mirror axis, where the point selected according to the point pair selection rule is the first point, and the mirror point corresponding to the first point is the second point, that is, the point pair contains the first point and the second point; then the pixel values ​​(i.e., grayscale values) of the two points in each point pair are compared, if the first point is greater than the second point, the value is 1, otherwise the value is 0, and the comparison values ​​of all point pairs are combined to obtain a set of feature values ​​composed of 1 and 0, and the feature value is the feature descriptor of point (i, j) located in the G channel image. In this embodiment, the length of the feature descriptor is 64-bit binary code, and the 64-bit binary code is converted into 8 decimal numbers (0 to 255) corresponding to 8 8-bit binary codes.

[0137] Step 4: Use the first channel feature, the second channel feature, and the third channel feature as the index result, establish the association relationship between the descriptor and the index result, and count the association relationship of each coordinate point to obtain the fourth dictionary model. Specifically, use the first channel feature, the second channel feature, and the third channel feature of the coordinate point (i, j) as the index result, associate the feature descriptor of the coordinate point (i, j) located in the G channel with the index result, store the association relationship between the feature descriptor and the index result, and build a feature descriptor index library.

[0138] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. An image fusion method based on a dictionary model, characterized in that: The following steps are involved: Step 1, respectively obtaining the image features of each coordinate point of the image to be fused in the three band images, which are the first band features, the second band features and the third band features; Step 2: for each coordinate point, obtain a first mapping ratio by indexing in the first dictionary model based on the first band feature, and obtain a first channel index value by calculating according to the first band feature and the first mapping ratio; Obtain the first channel feature according to the first channel index value; Based on the second band feature and the first channel feature, a second mapping ratio is obtained by indexing in the second dictionary model, and a second channel index value is calculated according to the second band feature and the second mapping ratio; A third mapping ratio is obtained by indexing in a third dictionary model based on the third band feature and the first channel feature, and a third channel index value is calculated according to the third band feature and the third mapping ratio; Step 3: Superimpose the first channel index value, the second channel index value, and the third channel index value of each coordinate point to obtain a fused true color image. 。 2. The image fusion method based on a dictionary model according to claim 1, characterized in that: For a single coordinate point, a local area around the coordinate point is selected based on a preset size, and the image feature is a statistical value of the grayscale of the local area.

3. The image fusion method based on a dictionary model according to claim 1, characterized in that: In step 2, if there is a value greater than 255 in the first channel index value of all coordinate points, normalization processing is performed and then multiplied by 255; Alternatively, if there is a value greater than 255 in the second channel index value of all coordinate points, normalization is performed and then multiplied by 255; Alternatively, if there is a value greater than 255 among the third channel index values ​​of all coordinate points, normalization processing is performed and then multiplied by 255.

4. The image fusion method based on a dictionary model according to claim 1, characterized in that: The method for constructing the first dictionary model, the second dictionary model and the third dictionary model in step 2 comprises the following steps: Step 2-1, obtaining image features of each coordinate point of the sample image in three band images, namely, first band features, second band features, and third band features; Obtain the image features of three channels of each coordinate point, which are the first channel feature, the second channel feature and the third channel feature; Step 2-2, for a single coordinate point, calculate a first mapping ratio between the first band feature and the first channel feature, use the first band feature as an index condition, establish an association relationship between the index condition and the first mapping ratio, and count the association relationship of each coordinate point to obtain a first dictionary model; For a single coordinate point, a second mapping ratio of the second channel feature to the first channel feature is calculated, the second band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the second mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a second dictionary model; For a single coordinate point, the third mapping ratio of the third channel feature to the first channel feature is calculated, the third band feature and the first channel feature are used as index conditions, an association relationship between the index condition and the third mapping ratio is established, and the association relationship of each coordinate point is counted to obtain a third dictionary model.

5. The image fusion method based on a dictionary model according to claim 4, characterized in that: Before step 2-1, the method further includes acquiring a band image and a true color image collected for the same target as a sample image; The image features of the three channels of each coordinate point obtained in step 2-1 are the image features of each coordinate point in the three channels of the true color image, which are the first channel features, the second channel features and the third channel features.

6. The image fusion method based on a dictionary model according to claim 4, characterized in that: In the step 2-2, the statistical association relationship of each coordinate point to obtain the first dictionary model specifically comprises: when the index conditions of the first mapping ratios of multiple coordinate points are the same, taking the average of the multiple first mapping ratios as the index result of the index condition; And / or, the statistical association relationship of each coordinate point to obtain the second dictionary model specifically comprises: when the index conditions of the second mapping ratios of multiple coordinate points are the same, taking the average of the multiple second mapping ratios as the index result of the index condition; And / or, the statistical association relationship of each coordinate point to obtain the third dictionary model is specifically: when the index conditions of the third mapping ratios of multiple coordinate points are the same, the average of the multiple third mapping ratios is used as the index result of the index condition.

7. The image fusion method based on a dictionary model according to claim 4, characterized in that: If more than three bands of images are obtained in step 2-1, the band images with similar bands are merged, and finally three images are retained, and image features are extracted from each of them.

8. The image fusion method based on a dictionary model according to claim 4, characterized in that: The step 2-1 of obtaining the image features of the three channels of each coordinate point specifically includes: Calculate the descriptor of each coordinate point in the selected band image; For a single coordinate point, the index results of the first channel feature, the second channel feature and the third channel feature are obtained by indexing in the fourth dictionary model based on the descriptor, and the index results are used as the image features of the three channels of the coordinate point.

9. The image fusion method based on a dictionary model according to claim 8, characterized in that: Obtain image features in three channels of each coordinate point of the true color image in the sample image, which are first channel features, second channel features and third channel features, and calculate the descriptor of each coordinate point in the selected channel; The first channel feature, the second channel feature and the third channel feature are used as index results, an association relationship between the descriptor and the index result is established, and the association relationship of each coordinate point is counted to obtain a fourth dictionary model.

10. The image fusion method based on dictionary model according to claim 9, characterized in that: The method of calculating the descriptor of each coordinate point includes: Step 3-1, calculate the centroid coordinates (x, y) of the image block centered at the coordinate point (i, j) and with a side length of m in the image according to the grayscale centroid method; Step 3-2, calculate the vector from the coordinate point (i, j) to the centroid coordinate (x, y), and the direction of the vector is used as the angle direction of the coordinate point (i, j); Step 3-3, calculating the descriptor of point (i, j) according to the angle direction and the grayscale information of the image block.

Images