A multi-resolution multi-spectral image fusion super-resolution method, system and medium

Abstract

The application discloses a kind of multi-resolution multispectral image fusion super-resolution method, system and medium, and the application includes obtaining the image I to be super-resolved in same scene and two guide images;The up-sampling image I of image I to be super-resolved is up-sampled as H ;I H And guide image are sliced;The average gray value of each slice of I H Is calculated, and according to average gray value, select one to be fused slice S H In two guide images;Image I to be super-resolved is sliced as to be fused slice I L ;To be fused slice S H , I L Fusion obtains the slice after super-resolution, and the final multispectral image of high resolution is obtained by splicing.This application aims to realize a kind of high quality, real-time good fusion super-resolution technology suitable for various multi-resolution multispectral images, to solve the problem that spatial resolution is not greatly improved, spectral distortion and real-time is not high in existing fusion super-resolution technology.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a multi-resolution, multispectral image fusion super-resolution method, system, and medium. Background Technology

[0002] In recent years, multispectral imaging technology has attracted considerable interest. Multispectral technology refers to the ability to simultaneously acquire multiple optical spectrum bands (usually three or more), extending beyond visible light into infrared and ultraviolet light. Compared to traditional red-green-blue three-channel visible light cameras, multispectral cameras can obtain information across more spectral bands. Multispectral imaging technology primarily utilizes various filters, beam splitters, and photosensitive elements to enable the camera to simultaneously receive information about the same object at different wavelengths. Multispectral imaging technology holds immense importance in the field of image acquisition and processing. Spatial resolution, as a measure of a sensor's ability to discern details in the output image, is a crucial technical indicator for image application value. Spatial resolution measures the detail of objects in a pixel-based image, representing image detail in pixels. Image spatial resolution is an important indicator of image quality; higher resolution indicates finer details, better quality, and richer information content. Image super-resolution technology, a branch of image quality enhancement, aims to recover high-resolution images from low-resolution images or image sequences. It is a modern image processing technology with high scientific research value and wide applications. The most direct way to obtain high-resolution images is to use high-resolution image sensors. However, due to limitations in sensor and optical device manufacturing processes and costs, this is difficult to achieve in many situations and large-scale deployments. Therefore, acquiring high-resolution images using existing equipment through image super-resolution technology has significant practical implications. With the rapid development of image processing technology in recent years, more and more image super-resolution methods adopt image fusion-based approaches. Image fusion aims to process multi-source data from time or space according to certain rules to obtain a more accurate and information-rich synthetic image. Image fusion-based super-resolution methods utilize high-resolution images to achieve super-resolution of low-resolution images within the same scene. Among the common fusion algorithms, the SFIM (Smoothing Filter-based Intensity Modulation) fusion algorithm, based on brightness smoothing filter adjustment, is a spatial-spectral combination fusion method. The SFIM fusion process is simple, with relatively low computational complexity, and it performs well compared to some classic algorithms; however, the improvement in image spatial resolution after SFIM fusion is not significant, and some spectral distortion exists. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a multi-resolution multispectral image fusion super-resolution method, system and medium to address the above-mentioned problems of the prior art. This invention aims to realize a high-quality and real-time fusion super-resolution technology applicable to various multi-resolution multispectral images, so as to solve the problems of limited spatial resolution improvement, spectral distortion and low real-time performance in the existing fusion super-resolution technology.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A multi-resolution, multispectral image fusion super-resolution method includes: S1, acquire the low-resolution super-resolution image I to be super-resolved in the same scene and two high-resolution guide images in different bands; S2, upsample the super-resolution image I to the size of the guide image to obtain the upsampled image I. H ; S3, use a rectangle of a specified size to upsample the image I. H And guide the image to be sliced; S4, calculate the upsampled image I H The average gray value of each slice is used to select a corresponding slice from the two guide images as the high-resolution slice to be fused, based on the average gray value and a preset gray value threshold λ. H ; S5, slice the super-resolution image I to be fused using a rectangle of a specified size to obtain a low-resolution slice I to be fused. L ; S6, high-resolution slices to be fused S H Low-resolution slices to be fused I L The fusion yields a super-resolution slice; S7 stitches together the super-resolution slices to obtain the final high-resolution multispectral image.

[0005] Optionally, before step S1, the method further includes screening two high-resolution guide images of different bands: For multiple high-resolution candidate guide images of different bands corresponding to the low-resolution super-resolution image I to be super-resolution in the same scene, each candidate guide image is first downsampled to the same size as the super-resolution image I to be super-resolution, and then the similarity between the super-resolution image I to be super-resolution and each candidate guide image is calculated, and the two candidate guide images with the highest similarity are selected as the two guide images selected.

[0006] Optionally, calculating the similarity between the super-resolution image I and each candidate guide image refers to calculating the structural similarity (SSIM) between the super-resolution image I and each candidate guide image.

[0007] Optionally, in step S4, a corresponding slice is selected from the two guide images as the high-resolution slice to be fused, based on the average gray value and a preset gray threshold λ. H include: , In the above formula, S1 and S2 represent two guiding images, gray represents the average gray value of the slice, and I max I represents the maximum grayscale value of the slice. min λ is the minimum grayscale value of the slice, and λ is the preset grayscale threshold.

[0008] Optionally, in the two guiding images, guiding image S1 has a shorter wavelength than guiding image S2.

[0009] Optionally, step S6 includes: S6.1, the high-resolution slice to be fused S H Downsampling with low-resolution slice I to be fused L Downsampling slices S of the same size L The low-resolution slices to be fused, I L and downsampling slice S L The fusion yields a low-resolution fused slice R. L ; S6.2, merge the low-resolution slice R L Upsampling is performed on the high-resolution slice S to be fused. H Fusion slices R after upsampling of the same size H ; S6.3, the upsampled fused slice R H and high-resolution slices to be fused S H The super-resolution slices are obtained by fusion.

[0010] Optionally, in step S6.1, a low-resolution fused slice R is obtained through fusion. L The function expression is: , In the above formula, Indicates low-resolution slice I to be fused L The pixel value at coordinate (i,j). S represents the downsampled slice S L The pixel value at coordinates (i,j) in the image.

[0011] Optionally, the functional expression for the super-resolution slice obtained by fusion in step S6.3 is: , In the above formula, F represents the super-resolution slice. R represents the fused slice after upsampling.H The pixel value at coordinate (i,j). S represents a high-resolution slice to be fused. H The pixel value at coordinate (i,j).

[0012] Furthermore, the present invention also provides a multi-resolution multispectral image fusion super-resolution system, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the multi-resolution multispectral image fusion super-resolution method.

[0013] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured by a microprocessor to execute the multi-resolution multispectral image fusion super-resolution method.

[0014] Compared with the prior art, the present invention has the following main advantages: 1. The super-resolution multispectral image obtained is of high quality. The SFIM algorithm uses only one high-resolution guide image, while this invention uses two high-resolution images from different bands as guide images. This is because high-resolution images from different bands have complementary information. We calculate the low-resolution image slices to determine which high-resolution image is better suited as the guide image for that slice. The final super-resolution image contains information from both guide images, thus improving the quality of the super-resolution multispectral image. Therefore, this invention is more rationally designed than the SFIM algorithm, with higher overall quality, achieving a better balance between improving spatial resolution and preserving spectral information. Subjective evaluation and objective analysis results are consistent.

[0015] 2. Good real-time performance. The method of this invention is a traditional fusion algorithm, which, compared to deep learning algorithms, requires no training. In traditional fusion algorithms, the method of this invention only uses dot multiplication and dot division operations, without complex formulas and a large number of parameters, thus simplifying the fusion process and reducing computational complexity. Therefore, compared with traditional image fusion-based super-resolution methods, this invention has a simpler fusion process and lower computational complexity, thus exhibiting a significant advantage in processing speed. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention. Detailed Implementation

[0017] To address the issues of limited spatial resolution improvement and spectral distortion in traditional fusion algorithms, this invention provides a multi-resolution, multispectral image fusion super-resolution method that utilizes high-resolution images from different spectral bands for image fusion and super-resolution. This method effectively solves the problems of low spatial resolution improvement, spectral distortion, and poor real-time performance in existing fusion super-resolution methods. Figure 1 As shown, the multi-resolution multispectral image fusion super-resolution method in this embodiment includes: S1, acquire the low-resolution super-resolution image I to be super-resolved in the same scene and two high-resolution guide images in different bands; S2, upsample the super-resolution image I to the size of the guide image to obtain the upsampled image I. H ; S3, use a rectangle of a specified size to upsample the image I. H And guide the image to be sliced; S4, calculate the upsampled image I H The average gray value of each slice is used to select a corresponding slice from the two guide images as the high-resolution slice to be fused, based on the average gray value and a preset gray value threshold λ. H ; S5, slice the super-resolution image I to be fused using a rectangle of a specified size to obtain a low-resolution slice I to be fused. L ; S6, high-resolution slices to be fused S H Low-resolution slices to be fused I L The fusion yields a super-resolution slice; S7 stitches together the super-resolution slices to obtain the final high-resolution multispectral image.

[0018] In this embodiment, before step S1, two high-resolution guide images of different bands are selected: For multiple high-resolution candidate guide images of different bands corresponding to the low-resolution super-resolution image I to be super-resolved in the same scene, each candidate guide image is first downsampled to the same size as the super-resolved image I. Then, the similarity between the super-resolved image I and each candidate guide image is calculated, and the two candidate guide images with the highest similarity are selected as the two guide images. Specifically, in this embodiment, the selected image data comes from Sentinel-2 data. The low-resolution band image I to be super-resolved is a B11 band image with a resolution of 20 meters (center wavelength: 1610nm) and a spatial resolution of 5490×5490. For the selection of high-resolution band images for guidance, firstly, bicubic interpolation downsampling is performed on the high-resolution band images (B02, B03, B04, B08 bands) to make their spatial resolution consistent with that of image I. Then, the two candidate guide images with the highest similarity between the super-resolution image I and each candidate guide image are selected as the two selected guide images. It should be noted that there are various known methods for calculating similarity; therefore, the two guide images selected can be selected according to the required similarity calculation method. For example, as an optional implementation, in this embodiment, calculating the similarity between the super-resolution image I and each candidate guide image means calculating the structural similarity index Measure (SSIM) between the super-resolution image I and each candidate guide image. The expression for the SSIM calculation function is: , In the above formula, SSIM(x,y) represents the calculation of the structural similarity SSIM between images x and y. and Let x and y represent the mean values ​​of the image, respectively. and Let x and y represent the variances of the image, respectively. Represents the covariance of the x and y graphs. and It is a constant. In addition, one of the following similarity measures—luminance similarity, contrast similarity, and another structural similarity measure—can be used as needed: , , , In the above formula, The brightness similarity between images x and y. The contrast similarity between images x and y. The structural similarity between images x and y. It is a constant. and These represent the standard deviations of images x and y, respectively. The meanings of the remaining symbols are the same as those in the expression for the SSIM structural similarity calculation function.

[0019] In this embodiment, the two high-resolution images with the highest structural similarity (SSIM) are selected as guide images. The final guide image S1 is a B04 band image with a resolution of 10 meters (center wavelength: 665nm), and the guide image S2 is a B08 band image with a center wavelength: 842nm. Both have a spatial resolution of 10980×10980.

[0020] In step S2 of this embodiment, the image to be super-resolved I is upsampled to the size of the guide image to obtain the upsampled image I. H Specifically, the low-resolution image I is upsampled by bicubic interpolation to obtain the upsampled image I. H This ensures that its spatial resolution is consistent with that of the high-resolution guide image. Specifically, in this embodiment, the super-resolution image I is a B11 band image with a spatial resolution of 5490×5490. It is upsampled using bicubic interpolation to make its spatial resolution 10980×10980.

[0021] In step S3 of this embodiment, a rectangle of a specified size is used to upsample the image I. H When slicing the guided image, the selected rectangle size is 20×20. The rectangle is used to traverse and slice the B11, B04, and B08 band images in sequence, resulting in 549×549 slices corresponding to the B11, B04, and B08 bands respectively.

[0022] In step S4, based on the average gray value and a preset gray threshold λ, a corresponding slice is selected from the two guide images as the high-resolution slice to be fused, S. H Based on whether the average gray value is greater than a preset gray value threshold λ, a corresponding slice can be selected from the two guide images as the high-resolution slice to be fused, S. H For example, if the average gray value is greater than the preset gray value threshold λ, slice S1 is selected; otherwise, slice S2 is selected. In this case, the gray value threshold λ should be the gray value (ranging from 0 to 255). However, considering that a static gray value threshold λ may lead to inaccurate selection of the guide image slice, this embodiment adopts a dynamic threshold selection method. The gray value threshold λ is used as a weighting coefficient, combined with the minimum and maximum gray values ​​of the slice to generate a dynamic threshold, thereby improving the accuracy of guide image slice selection. Specifically, in step S4 of this embodiment, based on the average gray value and the preset gray value threshold λ, a corresponding slice is selected from the two guide images as the high-resolution slice to be fused, S.H include: , In the above formula, S1 and S2 represent two guide images, gray represents the average gray value of the slice, and I max I represents the maximum grayscale value of the slice. min λ is the minimum grayscale value of the slice, and λ is the preset grayscale threshold.

[0023] Experiments have shown that super-resolution of the guide image with a shorter wavelength is clearer in areas with high grayscale values. Therefore, as a preferred implementation, in this embodiment, guide image S1 has a shorter wavelength than guide image S2, thereby obtaining a clearer guide image slice.

[0024] In step S5 of this embodiment, when slicing the super-resolution image I using a rectangle of a specified size, the size of the rectangle can be the same as or different from that in step S3. For example, as an optional implementation, the rectangle size selected in this embodiment is 10×10. The rectangle is used to perform a traversal slicing operation on the B11 band super-resolution image I, resulting in 549×549 B11 band slices I to be fused. L .

[0025] In this embodiment, step S6 includes: S6.1, the high-resolution slice to be fused S H Downsampling with low-resolution slice I to be fused L Downsampling slices S of the same size L The low-resolution slices to be fused, I L and downsampling slice S L The fusion yields a low-resolution fused slice R. L ; S6.2, merge the low-resolution slice R L Upsampling is performed on the high-resolution slice S to be fused. H Fusion slices R after upsampling of the same size H ; S6.3, the upsampled fused slice R H and high-resolution slices to be fused S H The super-resolution slices are obtained by fusion.

[0026] In this embodiment, step S6.1 yields a low-resolution fused slice R. L The function expression is: , In the above formula, Indicates low-resolution slice I to be fused LThe pixel value at coordinate (i,j). S represents the downsampled slice S L The pixel value at coordinates (i,j) in the image.

[0027] In this embodiment, the functional expression for the super-resolution slice obtained by fusion in step S6.3 is: , In the above formula, F represents the super-resolution slice. R represents the fused slice after upsampling. H The pixel value at coordinate (i,j). S represents a high-resolution slice to be fused. H The pixel value at coordinate (i,j).

[0028] The multi-resolution multispectral image fusion super-resolution method in this embodiment is specifically implemented in the PyCharm platform integrated development environment, combined with the open-source OpenCV machine vision image processing library. The algorithm code is written in Python, and this software can run stably on Windows 7 and above operating systems, including embedded systems. To verify the multi-resolution multispectral image fusion super-resolution method of this embodiment, the multispectral images stitched together by this method were validated, and the results are shown in Table 1.

[0029] Table 1: Evaluation results of objective indicators for multispectral images obtained by super-resolution.

[0030] In Table 1, SSIM refers to the structural similarity between the super-resolution multispectral image and the actual multispectral image. Generally, the SSIM range for this method is 0.8-0.85, the peak signal-to-noise ratio ranges from 25-30 dB, and the processing time is 30-200 s. As shown in Table 1, the multispectral image fusion super-resolution method in this embodiment exhibits good SSIM and PSNR values, and the processing time for the entire image is short. Furthermore, actual testing on Sentinel-2 data demonstrates that this method improves the spatial detail of the fused image while preserving multispectral information as much as possible, and can achieve super-resolution quickly, demonstrating high practical application value. Therefore, the multi-resolution multispectral image fusion super-resolution method in this embodiment offers high super-resolution quality, good real-time performance, and is suitable for fusion super-resolution of various multi-resolution multispectral images. It effectively solves the problems of limited spatial resolution improvement, spectral distortion, and low real-time performance in existing fusion super-resolution techniques.

[0031] Furthermore, this embodiment also provides a multi-resolution multispectral image fusion super-resolution system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the multi-resolution multispectral image fusion super-resolution method. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the multi-resolution multispectral image fusion super-resolution method.

[0032] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0033] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-resolution, multispectral image fusion super-resolution method, characterized in that, include: S1, acquire the low-resolution super-resolution image I to be super-resolved in the same scene and two high-resolution guide images in different bands; S2, upsample the super-resolution image I to the size of the guide image to obtain the upsampled image I. H ; S3, use a rectangle of a specified size to upsample the image I. H And guide the image to be sliced; S4, calculate the upsampled image I H The average gray value of each slice is used to select a corresponding slice from the two guide images as the high-resolution slice to be fused, based on the average gray value and a preset gray value threshold λ. H : ， In the above formula, S1 and S2 represent two guide images, gray represents the average gray value of the slice, and I max I represents the maximum grayscale value of the slice. min λ is the minimum grayscale value of the slice, and λ is the preset grayscale threshold. S5, slice the super-resolution image I to be fused using a rectangle of a specified size to obtain a low-resolution slice I to be fused. L ; S6, high-resolution slices to be fused S H Low-resolution slices to be fused I L The super-resolution slices are obtained by fusion; S7 stitches together the super-resolution slices to obtain the final high-resolution multispectral image.

2. The multi-resolution multispectral image fusion super-resolution method according to claim 1, characterized in that, Before step S1, two high-resolution guide images of different bands are selected: For multiple high-resolution candidate guide images of different bands corresponding to the low-resolution super-resolution image I in the same scene, each candidate guide image is first downsampled to the same size as the super-resolution image I, and then the similarity between the super-resolution image I and each candidate guide image is calculated. The two candidate guide images with the highest similarity are selected as the two guide images.

3. The multi-resolution multispectral image fusion super-resolution method according to claim 2, characterized in that, The calculation of the similarity between the super-resolution image I and each candidate guide image refers to the calculation of the structural similarity (SSIM) between the super-resolution image I and each candidate guide image.

4. The multi-resolution multispectral image fusion super-resolution method according to claim 1, characterized in that, Of the two guiding images, guiding image S1 has a shorter wavelength than guiding image S2.

5. The multi-resolution multispectral image fusion super-resolution method according to claim 1, characterized in that, Step S6 includes: S6.1, the high-resolution slice to be fused S H Downsampling with low-resolution slice I to be fused L Downsampling slices S of the same size L The low-resolution slices to be fused, I L and downsampling slice S L The fusion yields a low-resolution fused slice R. L ; S6.2, merge the low-resolution slice R L Upsampling is performed on the high-resolution slice S to be fused. H Fusion slices R after upsampling of the same size H ; S6.3, the upsampled fused slice R H and high-resolution slices to be fused S H The super-resolution slices are obtained by fusion.

6. The multi-resolution multispectral image fusion super-resolution method according to claim 5, characterized in that, In step S6.1, a low-resolution fused slice R is obtained through fusion. L The function expression is: ， In the above formula, Indicates low-resolution slice I to be fused L The pixel value at coordinate (i,j). S represents the downsampled slice S L The pixel value at coordinates (i,j) in the image.

7. The multi-resolution multispectral image fusion super-resolution method according to claim 5, characterized in that, The functional expression for the super-resolution slice obtained by fusion in step S6.3 is: ， In the above formula, F represents the super-resolution slice. R represents the fused slice after upsampling. H The pixel value at coordinate (i,j). S represents a high-resolution slice to be fused. H The pixel value at coordinate (i,j).

8. A multi-resolution, multispectral image fusion super-resolution system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the multi-resolution multispectral image fusion super-resolution method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to perform the multi-resolution multispectral image fusion super-resolution method according to any one of claims 1 to 7.

Images