An image fusion method, device and computer readable storage medium

Abstract

The application discloses an image fusion method, device and computer readable storage medium. The image fusion method comprises the following steps: extracting high-frequency components and low-frequency components of a first image to obtain first high-frequency components and first low-frequency components; extracting high-frequency components and low-frequency components of a second image to obtain second high-frequency components and second low-frequency components; wherein the resolution of the first image is lower than that of the second image; determining the edge intensity of each pixel position in the second high-frequency components, and determining the local difference measure value of each pixel position in the high-frequency components of the first image and the second image; performing pixel-by-pixel weighted fusion on the first high-frequency components and the second high-frequency components to obtain fused high-frequency components; and obtaining a fused image based on the fused high-frequency components and the fused low-frequency components. Through the above method, the application can solve the problem of loss of edge details in the fused image when the current image fusion method is used for image fusion.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to an image fusion method, apparatus and computer-readable storage medium. Background Technology

[0002] Image fusion has become a research hotspot in the field of image engineering in recent years. Specifically, image fusion refers to a new technology that integrates images from different sources. Its purpose is to extract information from multiple source images and combine the information from two or more source images to obtain a more accurate, comprehensive, and reliable image description of the same scene or target, thereby improving the reliability of target recognition.

[0003] Existing technologies for image fusion typically result in some loss of spectral information in the original images, causing the fused image to lose some subtle details such as edges and textures, thus leading to poor image quality. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide an image fusion method, device and computer-readable storage medium that can solve the problem of loss of edge detail information in the fused image when using current image fusion methods.

[0005] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: providing an image fusion method, the method comprising: extracting high-frequency components and low-frequency components of a first image to obtain a first high-frequency component and a first low-frequency component; extracting high-frequency components and low-frequency components of a second image to obtain a second high-frequency component and a second low-frequency component; wherein, the resolution of the first image is lower than the resolution of the second image; determining the edge intensity of each pixel position in the second high-frequency component; determining the local difference metric value of each pixel position in the high-frequency components of the first image and the second image respectively, obtaining a first local difference metric value and a second local difference metric value of each pixel position; weighting and fusing the first high-frequency component and the second high-frequency component pixel by pixel to obtain a fused high-frequency component; obtaining a fused image based on the fused high-frequency component and the fused low-frequency component; wherein, the fused low-frequency component is obtained by fusing the first low-frequency component and the second low-frequency component; wherein, in the process of pixel-by-pixel weighted fusion, the weight of each pixel position in the first high-frequency component is positively correlated with the first local difference metric value of each pixel position, and the weight of each pixel position in the second high-frequency component is positively correlated with the second local difference metric value and the edge intensity of each pixel position.

[0006] In one embodiment, the weight of each pixel position in the first high-frequency component is the ratio of the first local difference metric value of each pixel position to a preset value; the weight of each pixel position in the second high-frequency component is the ratio of the product of the second local difference metric value and the edge intensity to a preset value, wherein the preset value is equal to the product of the second local difference metric value and the edge intensity plus the first local difference metric value.

[0007] In one embodiment, determining the edge intensity of each pixel location in the second high-frequency component includes: processing a first local region centered on each pixel location in the second image using a gradient operator to obtain the edge intensity of each pixel location.

[0008] In one embodiment, determining the local difference metric for each pixel location in the high-frequency components of the first image and the second image includes: calculating the local variance of a second local region centered on each pixel location in the first image to obtain a first local difference metric for each pixel location; and calculating the local variance of a third local region centered on each pixel location in the second image to obtain a second local difference metric for each pixel location.

[0009] In one embodiment, a fused image is obtained based on the fusion of high-frequency components and the fusion of low-frequency components. The preceding steps include: using a first low-frequency component as a reference, performing histogram matching on a second low-frequency component to obtain a histogram-matched second low-frequency component; and fusing the histogram-matched second low-frequency component and the first low-frequency component to obtain a fused low-frequency component.

[0010] In one embodiment, fusing the second low-frequency component and the first low-frequency component after histogram matching to obtain a fused low-frequency component includes: calculating the local region energy of each pixel position of the first low-frequency component and the second low-frequency component after histogram matching, so as to obtain the first local region energy and the second local region energy of each pixel position respectively; using the pixel value of each first pixel position in the first low-frequency component as the pixel value of each first pixel position in the fused low-frequency image, wherein the first local region energy of each first pixel position is greater than the second local region energy of each first pixel position; and using the pixel value of each second pixel position in the first low-frequency component as the pixel value of each second pixel position in the fused low-frequency image, wherein the first local region energy of each second pixel position is greater than the second local region energy of each second pixel position.

[0011] In one embodiment, calculating the local region energy at each pixel location of the first low-frequency component and the second low-frequency component after histogram matching includes: calculating the local average gray value at each pixel location of the first low-frequency component and the second low-frequency component after histogram matching, and obtaining the first local region energy and the second local region energy at each pixel location respectively.

[0012] In one embodiment, the first image is the luminance component of a multispectral remote sensing image, and the second image is a panchromatic remote sensing image; a fused image is obtained based on the fusion of high-frequency components and the fusion of low-frequency components, including: obtaining a fused luminance component based on the fusion of high-frequency components and the fusion of low-frequency components; replacing the luminance component in the multispectral remote sensing image with the fused luminance component to obtain the fused multispectral remote sensing image.

[0013] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is to provide a remote sensing image fusion device, including a processor, which is used to execute instructions to implement the image fusion method as described above.

[0014] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is to provide a computer-readable storage medium for storing instruction / program data, which can be executed to implement the image fusion method as described above.

[0015] The beneficial effects of this invention are as follows: Unlike existing technologies, this invention extracts high-frequency and low-frequency components from a first image to obtain a first high-frequency component and a first low-frequency component; it also extracts high-frequency and low-frequency components from a second image to obtain a second high-frequency component and a second low-frequency component; wherein the resolution of the first image is lower than that of the second image; the edge intensity of each pixel in the second high-frequency component is determined, and the local difference metric value of each pixel in the high-frequency components of the first and second images is determined. Based on this, the first and second high-frequency components are weighted and fused pixel-by-pixel to obtain a fused high-frequency component; and a fused image is obtained based on the fused high-frequency component and the fused low-frequency component. By using the edge intensity of each pixel in the second high-frequency component as the weight of the high-frequency component of the second image for weighted fusion of the high-frequency components, the clarity of edge details in the fused image is effectively increased while reducing the distortion of spectral information in the fused image. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0017] Figure 1 This is a flowchart illustrating one embodiment of the image fusion method of the present invention;

[0018] Figure 2 This is a flowchart illustrating one embodiment of low-frequency component fusion of the present invention;

[0019] Figure 3 It is a global image and a magnified view of a local area of ​​a multispectral remote sensing image;

[0020] Figure 4 It is a panchromatic remote sensing image of the entire image and a magnified local area;

[0021] Figure 5 This is a schematic diagram of the fused multispectral image of the present invention;

[0022] Figure 6 This is a schematic diagram of one embodiment of the image fusion device of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image fusion method of the present invention, which includes:

[0025] Step 101: Extract the high-frequency and low-frequency components of the first image to obtain the first high-frequency component and the first low-frequency component; extract the high-frequency and low-frequency components of the second image to obtain the second high-frequency component and the second low-frequency component.

[0026] Optionally, the first image can be transformed to extract its high-frequency and low-frequency components, thereby obtaining the first high-frequency component and the first low-frequency component. That is, the first high-frequency component is the high-frequency component of the first image, and the first low-frequency component is the low-frequency component of the first image.

[0027] The second image is then transformed to extract its high-frequency and low-frequency components, resulting in a second high-frequency component and a second low-frequency component. Specifically, the second high-frequency component is the high-frequency component of the second image, and the second low-frequency component is the low-frequency component of the second image.

[0028] The methods for transforming the first and second images are not limited to non-subsampled contourlet transform (NSCT) or non-subsampled shearlet transform (NSST), as long as they can extract the high-frequency and low-frequency components in the image.

[0029] Alternatively, low-pass filtering can be applied to the first image and the second image separately to extract the high-frequency and low-frequency components of the first image and the second image, thereby obtaining the first high-frequency component, the first low-frequency component, the second high-frequency component, and the second low-frequency component. Low-pass filtering can be performed on the image using multiple low-pass filtering layers.

[0030] Alternatively, the pyramid transform method can be used to transform the first image, extracting its high-frequency and low-frequency components to obtain the first high-frequency component and the first low-frequency component; and the second image can be transformed to extract its high-frequency and low-frequency components to obtain the second high-frequency component and the second low-frequency component. Specifically, the main steps of the pyramid transform method can include low-pass filtering, upsampling, downsampling, and band-pass filtering. Each pyramid transform divides the image into a level. For example, using the second image as the input image for the pyramid transform, after the pyramid transform, the second high-frequency component and the second low-frequency component of the corresponding level of the second image can be obtained.

[0031] In a specific example, three pyramid transformations can be performed on both the first and second images, dividing them into three levels. For each level of the first and second images, a first high-frequency component, a first low-frequency component, a second high-frequency component, and a second low-frequency component are extracted. For instance, using the second image as the input image for the three pyramid transformations, the first level of the second image yields the second high-frequency and second low-frequency components. These first-level second high-frequency and second low-frequency components are then used as the input image for the second pyramid transformation, and so on. After three pyramid transformations, the second image has three levels of second high-frequency and second low-frequency components. Subsequent image fusion based on the low-frequency and high-frequency components from multiple levels can achieve clear display of image details.

[0032] Before extracting the high-frequency and low-frequency components of the first image and the second image, the first and second images can be acquired. The resolution of the first image is lower than that of the second image.

[0033] In one feasible implementation, the first image is the luminance component of a visible light image, and the second image is an infrared image.

[0034] In another feasible implementation, the first image is the luminance component of a multispectral remote sensing image, and the second image is a panchromatic remote sensing image.

[0035] Optionally, when the first image is the brightness component of a multispectral remote sensing image and the second image is a panchromatic remote sensing image, the original satellite remote sensing image can be obtained before extracting the high-frequency and low-frequency components of the first image and the high-frequency and low-frequency components of the second image; the multispectral and panchromatic remote sensing images in the original satellite remote sensing image are preprocessed to obtain the registered multispectral and panchromatic remote sensing images.

[0036] When the first image is the brightness component of a multispectral remote sensing image, obtaining the first image can be achieved by extracting the brightness component from the multispectral remote sensing image.

[0037] Preferably, the first image can be obtained by performing a GIHS transform on the multispectral remote sensing image to extract the luminance component. The GIHS transform is a method based on the General Intensity Hue Saturation (IHS) spatial transform. Furthermore, the chromaticity and saturation components obtained from the GIHS transform of the multispectral remote sensing image can be retained for subsequent inverse transforms. Compared to traditional IHS transform methods that can only transform three channels of an image, the GIHS transform method used in this invention is applicable to multi-channel images and retains the characteristic of the IHS transform not changing the spectral line shape.

[0038] Optionally, the transformation formula for GIHS can be as follows:

[0039]

[0040] Among them, C N The Nth channel of the multispectral remote sensing image, I and V1~V N-1 For each component after the GIHS transformation, T N This is the transformation matrix in the GIHS transformation.

[0041] And calculate T N The formula is as follows:

[0042]

[0043] Where T N The specific calculation method is as follows And so on.

[0044] In other embodiments, multispectral remote sensing images can also be transformed using color space algorithms. For example, the RGB2YUV conversion formula can be applied to separate the luminance and chrominance in a multispectral remote sensing image to obtain a first image. Alternatively, multispectral remote sensing images can be transformed in other ways, as long as the separation of luminance and chrominance can be achieved.

[0045] Considering the potential errors that may occur during the acquisition of the first and second images, resulting in inaccurate recording of surface information and reduced image data quality, thus affecting the accuracy of image fusion, a correction process can be performed on the first and second images before extracting their high-frequency and low-frequency components. This correction addresses distortions, warps, blurring, and noise generated during image acquisition, ensuring the accuracy of image fusion. For example, bicubic interpolation can be used for upsampling to maintain consistent dimensions between the acquired first and second images.

[0046] Step 102: Determine the edge intensity of each pixel location in the second high-frequency component.

[0047] The edge intensity of each pixel position in the second high-frequency component can be determined so that the edge intensity of each pixel position in the second high-frequency component can be used to perform pixel-by-pixel weighted fusion of the first high-frequency component and the second high-frequency component.

[0048] In one embodiment, the edge intensity of each pixel in the second high-frequency component can be calculated using a gradient operator. Specifically, a gradient operator can be used to process a first local region centered on each pixel position in the second high-frequency component to obtain the edge intensity of each pixel position in the second high-frequency component. The gradient operator can be the Sobel operator, the Roberts Cross Edge Detector, etc., and its type is not limited.

[0049] Since the gradient value of each pixel is related to the image's edges, textures, and other details, the larger the gradient value of each pixel, the more prominent the image's edges, textures, and other details, indicating richer spatial information and a stronger sense of depth. Therefore, by using the edge strength of each pixel in the second high-frequency component and its positively correlated weights to perform pixel-by-pixel weighted fusion of the second and first high-frequency components, the edge and other details in the second high-frequency component can be preserved and highlighted in the fused high-frequency component as much as possible, effectively increasing the clarity of edge details in the fused image.

[0050] In another embodiment, the edge intensity (i.e., the gradient value of each pixel) of the second image can be calculated using a gradient operator. Optionally, the gradient operator can be used to process a first local region centered on each pixel location in the second image to obtain the edge intensity at each pixel location.

[0051] Since high-frequency components inherently represent drastically changing details such as edges, extracting edge intensity from high-frequency components will result in the loss of some detail information. Compared to directly extracting edge intensity from the second high-frequency component, this implementation method can effectively reduce the loss of detail information such as edges.

[0052] In another implementation, the edge intensity of each pixel in the second image can be directly determined using an edge detection algorithm. For example, when the second image is a panchromatic remote sensing image, each pixel in the panchromatic remote sensing image is filtered using an edge detection algorithm to obtain the edge intensity of each pixel. Compared to directly performing edge detection on the high-frequency components of the panchromatic remote sensing image, this effectively reduces the loss of detail information such as edges. The calculation formula for the edge detection algorithm can be:

[0053]

[0054] Where λ and ε are constants. The derivative of the panchromatic remote sensing image.

[0055] Thus, by obtaining the edge intensity of each pixel position through the method in this embodiment, the information injection of non-edge regions in the high-frequency components of the panchromatic remote sensing image can be constrained, the distortion of spectral information in the multispectral image can be reduced, and the effect of subsequent image fusion processing can be improved.

[0056] Step 103: Determine the local difference metric value of each pixel position in the high frequency components of the first image and the second image respectively, and obtain the first local difference metric value and the second local difference metric value of each pixel position respectively.

[0057] Specifically, the local difference metric value of each pixel position in the first high-frequency component of the first image is calculated to obtain the first local difference metric value of each pixel position; the local difference metric value of each pixel position in the second high-frequency component of the second image is calculated to obtain the second local difference metric value of each pixel position, so as to facilitate the subsequent calculation of weights based on each local difference metric value.

[0058] In one embodiment, the local difference measure can be calculated using local variance. Specifically, the local variance of each pixel location in the first high-frequency component of the first image can be calculated to obtain the first local variance of each pixel location; the local variance of each pixel location in the second high-frequency component of the second image can be calculated to obtain the second local variance of each pixel location. Alternatively, the local difference measure can be characterized using local standard deviation, local covariance, etc.

[0059] It should be noted that there is no restriction on the execution order of steps 102 and 103. Step 102 can be executed first and then step 103, or step 103 can be executed first and then step 102, or steps 102 and 103 can be executed simultaneously. All of these are reasonable and will not affect the subsequent calculation of the fusion of high-frequency components and the fusion of low-frequency components.

[0060] Step 104: Pixel-wise weighted fusion of the first high-frequency component and the second high-frequency component is performed to obtain the fused high-frequency component.

[0061] Specifically, based on the edge intensity of each pixel position in the second high-frequency component obtained in the above steps, the first local difference metric value of each pixel position in the first high-frequency component in the first image, and the second local difference metric value of each pixel position in the second high-frequency component in the second image, each pixel in the first high-frequency component and the second high-frequency component is weighted and fused one by one to obtain the fused high-frequency component. In the pixel-by-pixel weighted fusion process, the weight of each pixel position in the first high-frequency component is positively correlated with the first local difference metric value of each pixel position, and the weight of each pixel position in the second high-frequency component is positively correlated with both the second local difference metric value and the edge intensity of each pixel position.

[0062] In the first high-frequency component, the weight of each pixel position is the ratio of the first local difference metric value to a preset value; in the second high-frequency component, the weight of each pixel position is the ratio of the product of the second local difference metric value and the edge intensity to a preset value, where the preset value is the product of the second local difference metric value and the edge intensity plus the first local difference metric value.

[0063] Optionally, the calculation formula for the fused high-frequency components is as follows:

[0064]

[0065] Wherein, Hf(i,j) is the pixel value at position (i,j) in the fused high-frequency component, Var_p(i,j) is the local difference metric of the second high-frequency component of the second image at position (i,j), Var_I(i,j) is the local difference metric of the first high-frequency component at position (i,j), Wp(i,j) is the edge intensity of the second high-frequency component at position (i,j), Hp(i,j) is the pixel value at position (i,j) in the second high-frequency component of the second image, and HI(i,j) is the pixel value at position (i,j) in the first high-frequency component of the first image.

[0066] Since the high-frequency components extracted from the image represent the detailed information of the image, this application uses edge intensity to determine the weight, which can restrict the injection of brightness information in the second image. This can enhance the clarity of image fusion while reducing the loss of spectral information, and can also fuse the edge information of the high-resolution image into the image as much as possible, so that the edge and other details in the final fused image are relatively clear.

[0067] Optionally, in addition to obtaining the fused high-frequency components based on the above steps, fused low-frequency components can also be obtained, so that a fused image can be obtained based on the fused high-frequency components and the fused low-frequency components in subsequent steps. The fused low-frequency components are obtained by fusing the first low-frequency component and the second low-frequency component.

[0068] For further details, please refer to Figure 2 , Figure 2 This is a flowchart illustrating one embodiment of the low-frequency component fusion of the present invention.

[0069] Step 201: Using the first low-frequency component as a reference, perform histogram matching on the second low-frequency component to obtain the histogram-matched second low-frequency component.

[0070] Optionally, the gray-level distributions of the first and second low-frequency components can be statistically analyzed first, and a gray-level histogram can be established. Further, a mapping relationship between the gray levels of the first and second low-frequency components can be established based on the gray-level histogram. Using the gray level of the first low-frequency component as a benchmark, the gray levels of the second low-frequency component are mapped one-to-one. This histogram matching of the second low-frequency component can reduce the spectral distortion of the image caused by the large overall gray-level difference between the second and first low-frequency components during the low-frequency component fusion process. Furthermore, compared to the scheme of performing histogram matching on the first and second images, it can avoid affecting the second high-frequency component, thus preventing the loss of detailed features such as edge information in the second high-frequency component.

[0071] Step 202: Fuse the second low-frequency component and the first low-frequency component after histogram matching to obtain the fused low-frequency component.

[0072] Specifically, the local region energy of each pixel location of the first low-frequency component is first calculated to obtain the first local region energy of each pixel location; then, the local region energy of each pixel location of the second low-frequency component after histogram matching is calculated to obtain the second local region energy of each pixel location. It should be noted that the "region" here can be a single pixel, or it can include a single pixel and its surrounding neighborhood.

[0073] For example, the formula for calculating the energy of a local region is as follows:

[0074]

[0075] Where w(x,y) is the weight, (N,M) is the region size, A represents the low-frequency component, and (,j) is the local region center.

[0076] Based on the above formula, the local average gray value of each pixel position of the first low-frequency component can be calculated to obtain the first local average gray value of each pixel position; and the local average gray value of each pixel position of the second low-frequency component after histogram matching can be calculated to obtain the second local average gray value of each pixel position.

[0077] Alternatively, the local region energy of each pixel location of the first low-frequency component can be obtained by using image modeling with a Markov random field, thus obtaining the first local region energy of each pixel location. The local region energy of each pixel location of the second low-frequency component after histogram matching can then be calculated to obtain the second local region energy.

[0078] After determining the first local region energy and the second local region energy at each pixel location, the first local region energy and the second local region energy at each pixel location can be compared, and the first low-frequency component and the second low-frequency component can be fused based on the comparison result.

[0079] Specifically, when the energy of the first local region at each first pixel location is greater than the energy of the second local region at each first pixel location, the pixel value of each first pixel location in the first low-frequency component is used as the pixel value of each first pixel location in the fused low-frequency image.

[0080] If the energy of the first local region at each second pixel location is greater than the energy of the second local region at each second pixel location, then the pixel value of each second pixel location in the first low-frequency component is used as the pixel value of each second pixel location in the fused low-frequency image.

[0081] For example, the formula for fusing low-frequency components is:

[0082]

[0083] Where Lp(i,j) is the pixel value at position (i,j) in the second low-frequency component after histogram matching, Lp_Area(i,j) is the energy of the second local region at position (i,j) in the second low-frequency component; LI(i,j) is the pixel value at position (i,j) in the first low-frequency component, LI_Area(i,j) is the energy of the first local region at position (i,j) in the first low-frequency component; and Lf(i,j) is the pixel value at position (i,j) in the fused low-frequency component.

[0084] Step 105: Obtain the fused image based on the fusion of high-frequency components and the fusion of low-frequency components.

[0085] Specifically, after obtaining the fused high-frequency component and the fused low-frequency component based on the above steps, a fused image can be obtained based on the fused high-frequency component and the fused low-frequency component.

[0086] In one implementation, the high-frequency and low-frequency components can be inversely transformed to obtain the fused image. The inverse transformation method is not limited to the reverse application of transformation methods such as Non-subsampled Contourlet Transform (NSCT) and Non-subsampled Shearlet Transform (NSST).

[0087] In one specific implementation, the high-frequency and low-frequency components can be inversely transformed to obtain a fused luminance component; then, a fused image is obtained based on the fused luminance component. Taking the luminance component of a multispectral remote sensing image as an example, and a panchromatic remote sensing image as an example: after obtaining the fused luminance component, the luminance component in the multispectral remote sensing image is replaced with the fused luminance component to obtain the fused multispectral remote sensing image. Specifically, the fused luminance component and all other components of the multispectral remote sensing image except for the luminance component can be fused to obtain the fused multispectral remote sensing image. The method for fusing the fused luminance component and all other components of the multispectral remote sensing image except for the luminance component can be the inverse application of transformation methods such as GIHS or IHS.

[0088] To better illustrate the image fusion method of this application, the following specific embodiments of the image fusion method are provided as examples:

[0089] 1. Spatial transformation of the multispectral remote sensing image is performed using GIHS transform to extract its brightness component (i.e., the first image). The transformation of the brightness component in GIHS space does not change the spectral line variations of individual pixels in the original multispectral remote sensing image. The GIHS transform formula is as follows:

[0090]

[0091] In the formula, C N The Nth channel of the multispectral remote sensing image, I and V1~V N-1 For each component after the GIHS transformation, T N T is the transformation matrix in the GIHS transformation. N The formula is as follows:

[0092]

[0093] Where T N The specific calculation method is as follows And so on.

[0094] 2. Perform nonsubsampled contourlet transform (NSCT) on the brightness component of the multispectral remote sensing image and the panchromatic remote sensing image respectively to obtain the corresponding high-frequency and low-frequency components.

[0095] 3. Considering that high-frequency components represent detailed information of the original image and low-frequency components represent global information, a histogram-constrained fusion rule is adopted for low-frequency components: A histogram matching method is used to convert the histogram of the low-frequency components of the panchromatic remote sensing image into the shape of the histogram of the brightness low-frequency components of the multispectral remote sensing image, reducing the impact of pixel value differences between the low-frequency components of the panchromatic and multispectral remote sensing images on low-frequency component fusion; then, the local region energy of the low-frequency components of the panchromatic and multispectral remote sensing images after histogram matching is compared, and the larger local region energy is selected as the fusion result. The fusion rule for high-frequency components is as follows: edge detection is performed on the undecomposed panchromatic remote sensing image to obtain the edge intensity at each pixel location in the panchromatic remote sensing image; weighted fusion is then performed based on the edge intensity and local difference metric value at each pixel location in the panchromatic remote sensing image.

[0096] The histogram matching method mainly involves: statistically analyzing the grayscale distribution of the low-frequency components of the panchromatic remote sensing image and the grayscale distribution of the brightness low-frequency components of the multispectral remote sensing image; establishing a mapping relationship between the grayscale of the low-frequency components of the panchromatic remote sensing image and the grayscale of the brightness low-frequency components of the multispectral remote sensing image based on the grayscale histogram; and mapping the grayscale of the low-frequency components of the panchromatic remote sensing image one by one to obtain the low-frequency components of the panchromatic remote sensing image after histogram matching.

[0097] The energy of a local region mainly involves calculating the average gray value of that region. The formula is as follows:

[0098]

[0099] Where w(x, y) is the weight, (N, M) is the region size, A represents the low-frequency component, and (i, j) is the local region center. By comparing the Area(i, j) of the low-frequency component of the panchromatic remote sensing image after histogram matching with the brightness low-frequency component of the multispectral remote sensing image, the low-frequency component fusion result is obtained. The low-frequency fusion formula is as follows:

[0100]

[0101] Where Lp(i,j) is the low-frequency component value at position (i,j) of the panchromatic remote sensing image after histogram matching, LI(i,j) is the low-frequency component value at position (i,j) of the I component, and Lf(i,j) is the low-frequency component value at position (i,j) after fusion.

[0102] Edge detection operators are used to filter the undecomposed panchromatic remote sensing image to obtain the edge intensity at each pixel location. This restricts the injection of information into non-edge regions in the high-frequency components of the panchromatic remote sensing image, reducing the distortion of spectral information in the multispectral remote sensing image. The edge detection operator is as follows:

[0103]

[0104] Where λ and ε are constants. is the derivative of the panchromatic image.

[0105] The high-frequency fusion formula is as follows:

[0106]

[0107] In the formula, Hf(i,j) represents the fused high-frequency component, Var_p(i,j) represents the local difference measure of the high-frequency component of the panchromatic remote sensing image at position (i,j), Var_I(i,j) represents the local difference measure of the brightness high-frequency component of the multispectral remote sensing image at position (i,j), and W... p (i,j) represents the edge intensity of each pixel in the panchromatic remote sensing image, Hp(i,j) represents the high-frequency component of the panchromatic remote sensing image, and HI(i,j) represents the high-frequency component of the brightness of the multispectral remote sensing image.

[0108] 4. Based on the high-frequency and low-frequency components obtained by fusion, perform NSCT inverse transform to obtain the fused luminance component, and then perform GIHS inverse transform to obtain the final multispectral fused image.

[0109] For an example of fusion, please refer to [link / reference]. Figure 3 , Figure 4 as well as Figure 5 . Figure 3 and Figure 4These are, respectively, the global image and magnified local area of ​​the unprocessed multispectral remote sensing image of the same scene, and the global image and magnified local area of ​​the panchromatic remote sensing image. Figure 5 This is the result of fusion after the embodiment of the present invention. It can be seen that the image processed by the embodiment of the present invention absorbs the hyperspectral information of multispectral remote sensing images and the high spatial resolution of panchromatic remote sensing images. The local features and details of the fused image are more prominent, and the clarity is higher.

[0110] Meanwhile, this invention prioritizes GIHS transform during image fusion. Compared to the traditional IHS transform method, which can only transform three channels of an image, the GIHS transform method is applicable to multi-channel images and retains the characteristic of IHS transform without changing the spectral line shape, thus improving the quality of the fused image. Furthermore, this invention addresses the characteristics of high-frequency components reflecting the detailed information of the original image and low-frequency components reflecting the global information of the original remote sensing image by designing different fusion rules for low-frequency and high-frequency components. In the low-frequency component fusion process: histogram matching is used to perform grayscale mapping transformation on the low-frequency components of the panchromatic remote sensing image, reducing the spectral distortion of the final fused image caused by the large overall grayscale difference between the panchromatic image's low-frequency components and the multispectral remote sensing image's brightness low-frequency components. In the high-frequency component fusion process, edge detection is performed on the undecomposed panchromatic remote sensing image using a detection operator to obtain the edge intensity at each pixel position in the panchromatic remote sensing image. This edge intensity is used as the weight for the high-frequency components in the weighted fusion of the panchromatic image, effectively increasing the clarity of edge details and reducing spectral distortion. In summary, the present invention can solve the problem of loss of detail information such as edges in the fused image when using current image fusion methods.

[0111] The image fusion method provided in this invention can be applied to electronic devices, such as image acquisition devices like cameras or mobile phones, personal computers (PCs), and tablets.

[0112] Please see Figure 6 , Figure 6 This is a schematic diagram of one embodiment of the image fusion device of the present invention. In this embodiment, the image fusion device includes a processor 50 and a memory 51 coupled to each other for cooperating to implement the image fusion method described in any of the above embodiments. The memory 51 also includes at least one computer program 52 running on the processor 50. When the processor 50 executes the computer program 52, it implements the steps in any of the above-described image fusion method embodiments.

[0113] The processor 50 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0114] The memory 51 can be a Smart Media Card (SMC), a Secure Digital Card (SD), a Flash Card, etc. Furthermore, the memory 51 can include both internal storage units and external storage devices. The memory 51 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of computer programs. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0115] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0116] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0118] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this invention.

Claims

1. An image fusion method, characterized in that, include: Extract the high-frequency and low-frequency components of the first image to obtain the first high-frequency component and the first low-frequency component; The high-frequency and low-frequency components of the second image are extracted to obtain the second high-frequency component and the second low-frequency component; wherein the resolution of the first image is lower than the resolution of the second image. Determine the edge intensity of each pixel location in the second high-frequency component; Determine the local difference metric value of each pixel position in the high frequency component of the first image and the second image respectively, and obtain the first local difference metric value and the second local difference metric value of each pixel position respectively. The local difference metric value represents one of the local variance, local standard deviation and local covariance of each pixel position in the first high frequency component of the first image and one of the local variance, local standard deviation and local covariance of each pixel position in the second high frequency component of the second image. The first high-frequency component and the second high-frequency component are weighted and fused pixel by pixel to obtain the fused high-frequency component; A fused image is obtained based on the fused high-frequency component and the fused low-frequency component; wherein the fused low-frequency component is obtained by fusing the first low-frequency component and the second low-frequency component; In the pixel-by-pixel weighted fusion process, the weight of each pixel position in the first high-frequency component is positively correlated with the first local difference metric of each pixel position, and the weight of each pixel position in the second high-frequency component is positively correlated with the second local difference metric of each pixel position and the edge intensity.

2. The image fusion method according to claim 1, characterized in that, The weight of each pixel position in the first high-frequency component is the ratio of the first local difference metric value of each pixel position to a preset value; the weight of each pixel position in the second high-frequency component is the ratio of the product of the second local difference metric value and the edge intensity to the preset value, wherein the preset value is equal to the product of the second local difference metric value and the edge intensity plus the first local difference metric value.

3. The image fusion method according to claim 1, characterized in that, Determining the edge intensity of each pixel location in the second high-frequency component includes: Using a gradient operator, the first local region centered at each pixel position in the second image is processed to obtain the edge intensity at each pixel position.

4. The image fusion method according to claim 1, characterized in that, The step of determining the local difference metric for each pixel position in the high-frequency components of the first image and the second image includes: Calculate the local variance of the second local region centered on each pixel position in the first image to obtain the first local difference metric value for each pixel position; Calculate the local variance of the third local region centered at each pixel location in the second image to obtain the second local difference metric value for each pixel location.

5. The image fusion method according to claim 1, characterized in that, The process of obtaining the fused image based on the fused high-frequency components and the fused low-frequency components includes the following steps: Using the first low-frequency component as a reference, histogram matching is performed on the second low-frequency component to obtain the histogram-matched second low-frequency component; The second low-frequency component after histogram matching is fused with the first low-frequency component to obtain the fused low-frequency component.

6. The image fusion method according to claim 5, characterized in that, The step of fusing the second low-frequency component after histogram matching with the first low-frequency component to obtain the fused low-frequency component includes: Calculate the local region energy at each pixel position of the first low-frequency component and the second low-frequency component after histogram matching, so as to obtain the first local region energy and the second local region energy at each pixel position respectively. The pixel value of each first pixel position in the first low-frequency component is used as the pixel value of each first pixel position in the fused low-frequency image, and the energy of the first local region of each first pixel position is greater than the energy of the second local region of each first pixel position; The pixel value of each second pixel position in the first low-frequency component is used as the pixel value of each second pixel position in the fused low-frequency image, and the energy of the first local region of each second pixel position is greater than the energy of the second local region of each second pixel position.

7. The image fusion method according to claim 6, characterized in that, The calculation of the local region energy at each pixel position of the first low-frequency component and the second low-frequency component after histogram matching includes: Calculate the local average gray value of each pixel position of the first low-frequency component and the second low-frequency component after histogram matching, and obtain the first local region energy and the second local region energy of each pixel position respectively.

8. The image fusion method according to claim 1, characterized in that, The first image is the brightness component of a multispectral remote sensing image, and the second image is a panchromatic remote sensing image; The fused image is obtained based on the fused high-frequency components and the fused low-frequency components, including: Based on the fused high-frequency component and the fused low-frequency component, the fused brightness component is obtained; The brightness component in the multispectral remote sensing image is replaced with the fused brightness component to obtain the fused multispectral remote sensing image.

9. An image fusion device, characterized in that, Includes a processor, the processor being configured to execute instructions to implement the image fusion method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instruction / program data that can be executed to implement the image fusion method as described in any one of claims 1-8.

Images