Optical satellite panchromatic image and multispectral image registration and fusion method and device

By dividing optical satellite full-color images and multispectral images into image registration groups, and using Fourier transform and voting parameter space to determine the target registration parameters, the high geometric accuracy fusion of optical satellite full-color images and multispectral images is achieved. It solves the problem that the image edges in traditional technology do not meet the mapping requirements, and improves the accuracy and efficiency of stereoscopic mapping.

CN118691654BActive Publication Date: 2024-11-22湖北省测绘工程院 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411165007.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-22
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

Traditional optical satellite full-color images and multi-spectral images cannot achieve registration and fusion with high geometric accuracy in stereoscopic mapping, resulting in the image edges not meeting the mapping requirements, and the existing technology cannot effectively solve this problem.

Method used

By dividing optical satellite full-color images and multi-spectral images into multiple image registration groups, the phase information is extracted using Fourier transform, the voting parameter space and correlation images are constructed, the target registration parameters are determined, and the coarse registration and pyramids are performed. Image matching to achieve high geometric accuracy image fusion.

Benefits of technology

It improves the accuracy and stability of image registration, reduces the impact of extreme values ​​on the fusion results, enhances the robustness of the algorithm, and improves the clarity and contrast of the image after fusion through feature screening and image enhancement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691654B_ABST
    Figure CN118691654B_ABST
Patent Text Reader

Abstract

The present invention provides a registration and fusion method and device for optical satellite panchromatic images and multispectral images, and relates to the field of image registration and fusion. The method includes: segmenting the optical satellite panchromatic images and multispectral images to obtain an image registration group; determining the first registration parameter corresponding to the image registration group according to the voting parameter space; determining the second registration parameter according to the first phase information and the second phase information; determining the registration parameter sequence according to the first registration parameter and the second registration parameter, and determining the target registration parameter from the registration parameter sequence; using the target registration parameter to roughly register the multispectral image to obtain a registered image, and fusing the optical satellite panchromatic image and the registered image to obtain a fused image. The present invention can not only improve the registration accuracy and realize the registration and fusion of high geometric accuracy in stereo mapping, but also reduce the influence of extreme values ​​on the fusion results, enhance the robustness of the algorithm to abnormal values, and make the final image registration more stable and reliable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image registration and fusion, and in particular to a registration and fusion method and device for optical satellite panchromatic images and multispectral images. Background Art

[0002] When traditional stereo mapping uses optical satellite images for stereo mapping, satellite panchromatic images are usually selected to ensure high geometric accuracy. However, the single color of panchromatic images limits the capture of color-related details and information during manual stereo mapping, which can easily lead to misunderstandings by operators and affect the accuracy of ground object judgment and work efficiency. Although the fusion of optical satellite panchromatic images and orthophotos of multispectral images can improve the accuracy of ground object judgment, it cannot be applied to stereo mapping.

[0003] Although the fusion of optical satellite panchromatic images and multispectral images can be applied to stereo mapping, it is impossible to achieve high-precision registration because the consistency of the spatial position of optical satellite panchromatic images and multispectral images cannot be guaranteed. Traditional fusion processing is to fuse the digital orthophoto map (DOM) corresponding to the corrected and registered panchromatic image with the DOM corresponding to the multispectral image. Although it can obtain a fused color image, the fused color image is a color DOM without parallax, which makes stereo mapping impossible.

[0004] How to apply high geometric precision registration and fusion images in stereo mapping, and use the fusion of optical satellite panchromatic images and multispectral images to obtain color satellite images with parallax and image edge preservation that can meet mapping needs, is a technical problem that needs to be solved urgently. However, there is currently no technical solution that can solve the above technical problems, and there is no registration and fusion method and device for optical satellite panchromatic images and multispectral images. Summary of the invention

[0005] The present invention provides a registration and fusion method and device for optical satellite panchromatic images and multispectral images, which divide the optical satellite panchromatic images and multispectral images into a plurality of image registration groups, fuse the two registration parameter determination methods, determine the target registration parameters for registration, thereby improving the registration accuracy of image registration, and further realizing registration and fusion with high geometric accuracy in stereo mapping.

[0006] In a first aspect, the present invention provides a registration and fusion method for optical satellite panchromatic images and multispectral images, comprising:

[0007] Segment the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image;

[0008] For each image registration group, a voting parameter space is constructed according to an edge point set of an optical satellite panchromatic sub-image and an edge point set of a multispectral sub-image, and a first registration parameter corresponding to the image registration group is determined according to the voting parameter space; first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image are extracted by Fourier transform, and a second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information;

[0009] Traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining an intermediate value of the registration parameter from the registration parameter sequence as a target registration parameter;

[0010] The multispectral image is roughly registered using the target registration parameters to obtain a registered image, and the optical satellite panchromatic image and the registered image are fused to obtain a fused image.

[0011] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the registration parameter sequence is determined according to all the first registration parameters and all the second registration parameters, and the intermediate value of the registration parameter is determined from the registration parameter sequence as the target registration parameter, including:

[0012] Sorting all first registration parameters and all second registration parameters in descending order to determine a registration parameter sequence;

[0013] Determining the intermediate value of the registration parameter as the target registration parameter from the registration parameter sequence comprises:

[0014] When m+n is an odd number, ;

[0015] When m+n is an even number, ;

[0016] In the equation, m is the total number of the first registration parameters, n is the total number of the second registration parameters, is the target registration parameter, is the registration parameter sequence, .

[0017] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the multispectral image is roughly registered by using the target registration parameter to obtain the registered image, including:

[0018]

[0019] in, is the position coordinate of the multispectral image, is the position coordinate of the image after registration, where and is the target registration parameter, , , , The preset rotation parameters.

[0020] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the multispectral image is roughly registered by using the target registration parameter to obtain the registered image, including:

[0021] Using the target registration parameters to coarsely register the multispectral image to obtain a coarsely registered image;

[0022] Extracting matching feature points from the coarsely registered image, intersecting the matching feature points with a preset digital elevation model through an image-to-ground forward solution algorithm, and determining the position coordinates of the feature points;

[0023] The position coordinates of the feature points are calculated by using a preset rational function model to solve the target coordinate positions of the matching feature points in the optical satellite panchromatic image;

[0024] The coarsely registered image is updated according to the position coordinates of the feature points to obtain the registered image, and the optical satellite panchromatic image is updated according to the target coordinate position, so as to use the updated optical satellite panchromatic image and the registered image to perform pyramid image matching step by step to obtain a fused image.

[0025] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the extracting matching feature points from the coarse registration image comprises:

[0026] For each pixel in the coarse registration image, the sum of squares of grayscale differences of adjacent pixels in each direction in the first preset window corresponding to the pixel is calculated, and the value with the minimum sum of squares is determined as the characteristic value of the pixel;

[0027] Traverse all pixel points and determine all pixel points with feature values ​​greater than a preset threshold as initial candidate points;

[0028] For each initial candidate point, a grayscale correlation coefficient corresponding to the initial candidate point is calculated according to the grayscale correlation between the initial candidate point and each pixel point in the first preset window, and initial candidate points whose grayscale correlation coefficients are less than a preset coefficient are eliminated to obtain intermediate candidate points;

[0029] Dividing the coarse registration image into a plurality of second preset windows, and for each second preset window, determining the middle candidate point with the largest eigenvalue in the second preset window as the target candidate point;

[0030] For each target candidate point, a third preset window is constructed with the target candidate point as the center, and the horizontal and vertical gradient values ​​of each pixel point in the third preset window are calculated to construct a gradient matrix; the gradient matrix is ​​processed by eigenvalue decomposition to obtain a first eigenvalue and a second eigenvalue, and a target candidate point whose target eigenvalue is greater than a preset eigenvalue is determined as the matching feature point;

[0031] The target eigenvalue is the minimum value of the first eigenvalue and the second eigenvalue.

[0032] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the grayscale correlation coefficient corresponding to the initial candidate point is calculated according to the grayscale correlation between the initial candidate point and each pixel point in the first preset window, including:

[0033]

[0034] in, is the grayscale correlation coefficient corresponding to any initial candidate point, is the grayscale of any pixel in the first preset window, is the grayscale mean value in the first preset window, for The derivative of , k is the number of rows of the first preset window, and l is the number of columns of the first preset window.

[0035] According to the registration and fusion method of the optical satellite panchromatic image and the multispectral image provided by the present invention, the fusing of the optical satellite panchromatic image and the registered image to obtain the fused image includes:

[0036] The registered image is processed by IHS transformation to obtain a brightness component, a hue component and a saturation component;

[0037] Performing histogram matching between the grayscale image corresponding to the optical satellite panchromatic image and the brightness component to determine the matched brightness component;

[0038] The IHS inverse transform processes the matched brightness component, hue component and saturation component to obtain the fused image.

[0039] According to the registration and fusion method of the optical satellite panchromatic image and the multispectral image provided by the present invention, after fusing the optical satellite panchromatic image and the registered image to obtain the fused image, the method further comprises:

[0040] Determining image features according to the spatial resolution, noise level and preset clarity of the fused image, and determining the degree of image sharpening according to the image features and preset sharpening parameters;

[0041] According to the image sharpening degree, determining a target Gaussian kernel from a correspondence between a preset sharpening degree and a preset Gaussian kernel;

[0042] Processing the fused image using the target Gaussian kernel to obtain a convolved image;

[0043] Performing pixel-by-pixel subtraction processing on the fused image and the convolved image to obtain a difference image;

[0044] Processing the difference image according to a preset sharpening strength coefficient to obtain a sharpened difference image;

[0045] The fused image and the sharpened difference image are superimposed pixel by pixel to obtain an image enhanced image.

[0046] According to the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention, the fused image is processed by using the target Gaussian kernel to obtain the convolved image, including:

[0047]

[0048] in, is any pixel value of the convolution image, is any pixel value in the neighborhood, is the size of the Gaussian kernel, is the standard deviation of the Gaussian kernel.

[0049] In a second aspect, a registration and fusion device for optical satellite panchromatic images and multispectral images is provided, comprising:

[0050] a segmentation unit, configured to segment the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image;

[0051] A determination unit is used to construct a voting parameter space for each image registration group according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image, and determine a first registration parameter corresponding to the image registration group according to the voting parameter space; extract first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image by Fourier transform, and determine a second registration parameter corresponding to the correlation image according to the first phase information and the second phase information;

[0052] A traversal unit, used for traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining a registration parameter intermediate value from the registration parameter sequence as a target registration parameter;

[0053] The fusion unit is used to perform coarse registration on the multispectral image using the target registration parameters to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image.

[0054] The present invention divides the optical satellite panchromatic image and the multispectral image into a plurality of image registration groups, and uses two registration parameter determination methods to determine the target registration parameters used for registration, thereby improving the registration accuracy of the image registration, and then realizing the registration fusion with high geometric accuracy in stereo mapping. Since the target registration parameter is the intermediate value of the registration parameter, the influence of the extreme value on the fusion result is reduced, and the robustness of the algorithm to abnormal values ​​is enhanced, making the final image registration more stable and reliable. The present invention also uses a multi-level feature screening method to extract matching feature points from the coarse registration image, greatly improving the accuracy, reliability and robustness of the matching feature point selection, and finally, by performing image enhancement on the fused image, the concept of image sharpening is introduced, and the target Gaussian kernel is more accurately determined, thereby improving the clarity and contrast of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 This is one of the flow diagrams of the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention;

[0057] Figure 2 This is the second flow chart of the registration and fusion method of optical satellite panchromatic image and multispectral image provided by the present invention;

[0058] Figure 3 It is a structural schematic diagram of the optical satellite panchromatic image and multispectral image registration and fusion device provided by the present invention;

[0059] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Figure 1 This is one of the flow diagrams of the registration and fusion method of the optical satellite panchromatic image and the multispectral image provided by the present invention, and the registration and fusion method of the optical satellite panchromatic image and the multispectral image comprises:

[0062] Step 101, segmenting the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image;

[0063] Step 102: for each image registration group, a voting parameter space is constructed according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image, and a first registration parameter corresponding to the image registration group is determined according to the voting parameter space; first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image are extracted by Fourier transform, and a second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information;

[0064] Step 103, traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining a registration parameter intermediate value from the registration parameter sequence as a target registration parameter;

[0065] Step 104: perform coarse registration on the multispectral image using the target registration parameters to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image.

[0066] In step 101, in order to improve the robustness of the target registration parameter registration, the present invention divides the optical satellite panchromatic image according to a preset grid to obtain all optical satellite panchromatic sub-images, and divides the multispectral image according to the preset grid to obtain all multispectral sub-images. For each subgrid in the preset grid, there is an optical satellite panchromatic sub-image and a multispectral sub-image, that is, an image registration group is constituted. The present invention first determines each image registration group corresponding to all subgrids.

[0067] In step 102, for each image registration group, step 102 is performed once. Specifically, a voting parameter space is constructed according to the edge point set of the optical satellite panchromatic sub-image and the edge point set of the multispectral sub-image, and the first registration parameter corresponding to the image registration group is determined according to the voting parameter space. Since the present invention does not register the orthophotos of the panchromatic satellite image and the multispectral fusion satellite image, but registers the panchromatic satellite image and the multispectral fusion satellite image, the registration accuracy needs to be considered when considering the registration. The same-name point is a correspondence between the feature extracted from the left reference image and the feature extracted from the right target image, that is, there is a one-to-one correspondence between several elements in the left set and several elements in the right set, and there is a spatial geometric constraint between the two images to be matched according to the imaging model, and the same-name point needs to satisfy the imaging geometric constraint. The present invention uses the possible correspondence between the elements in the left and right sets to vote, and the cumulative parameter space of the vote is the parameter space composed of the imaging geometric constraint parameters between the images. The goal is to determine the first registration parameter between the satellite images, and the optimal translation amount is determined in the parameter space by the voting method.

[0068] Specifically, the present invention uses the Canny operator to extract edge points from the optical satellite panchromatic sub-image and the multispectral sub-image, respectively, and each edge point contains pixel coordinates and edge direction. The edge point is projected, and two sets are constructed, A is the edge point set in the optical satellite panchromatic sub-image, and B is the edge point set in the multispectral sub-image. The voting parameter space is established, the horizontal axis represents the registration parameter b0, and the vertical axis represents the registration parameter a0. The translation amount is calculated for each possible correspondence, and 1 is accumulated in the corresponding grid in the parameter space. Finally, the translation amount of the cell corresponding to the peak in the cumulative space is determined as the first registration parameter. By voting and accumulating in the parameter space, the best first registration parameter can be effectively found, thereby realizing the coarse matching positioning between satellite images. Obtaining the first registration parameter can help improve the accuracy and stability of the registration, and provide a reliable basis for subsequent registration and fusion operations. In order to weaken the systematic error of the rational function coefficient (RPC) parameter of the multispectral image and make the relative position relationship between the optical satellite panchromatic image and the multispectral image more accurate, the present invention introduces a voting algorithm to correct the initial predicted position and register it, so that the initial position accuracy reaches about 20 to 30 pixels.

[0069] Optionally, the present invention converts the optical satellite panchromatic sub-image and the multispectral sub-image into the frequency domain through Fourier transform, and then calculates the phase correlation between them, and determines the translation amount between the images by maximizing the phase correlation.

[0070] Specifically, a two-dimensional Fourier transform is performed on the optical satellite panchromatic sub-image and the multispectral sub-image to convert them into the frequency domain. The Fourier transform can convert the image from the spatial domain to the frequency domain and represent the image as a superposition of frequency components. Then, phase information is extracted from the spectrum after the Fourier transform. The phase information reflects the relative position relationship between components of different frequencies in the image, that is, the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image are extracted by Fourier transform. Then, the correlation between the frequency domain phases of the first phase information and the second phase information is calculated. The phase correlation can be obtained by calculating the product of the two spectrum phases, that is, the product of the two spectrum phases is inverse Fourier transformed to obtain a correlation image, and the maximum value is found in the correlation image. The position corresponding to the maximum value is the optimal translation amount, that is, the second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information.

[0071] Optionally, the position of the maximum value indicates that the phases of the optical satellite panchromatic sub-image and the multispectral sub-image are most consistent at this position, that is, the best matching position. The present invention realizes accurate image registration based on the acquired second registration parameter, which is particularly suitable for image registration scenarios requiring high precision and high stability. Its advantage also lies in its strong robustness to noise and brightness changes, and can effectively deal with some complex image registration problems.

[0072] In step 103, the present invention traverses all image registration groups to determine all first registration parameters and all second registration parameters, and determines a registration parameter sequence according to all first registration parameters and all second registration parameters, and determines a registration parameter intermediate value from the registration parameter sequence as a target registration parameter, including:

[0073] Sorting all first registration parameters and all second registration parameters in descending order to determine a registration parameter sequence;

[0074] Determining the intermediate value of the registration parameter as the target registration parameter from the registration parameter sequence comprises:

[0075] When m+n is an odd number, in order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Figure 1This is one of the flow diagrams of the registration and fusion method of the optical satellite panchromatic image and the multispectral image provided by the present invention, and the registration and fusion method of the optical satellite panchromatic image and the multispectral image comprises:

[0077] Step 101, segmenting the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image;

[0078] Step 102: for each image registration group, a voting parameter space is constructed according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image, and a first registration parameter corresponding to the image registration group is determined according to the voting parameter space; first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image are extracted by Fourier transform, and a second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information;

[0079] Step 103, traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining a registration parameter intermediate value from the registration parameter sequence as a target registration parameter;

[0080] Step 104: perform coarse registration on the multispectral image using the target registration parameters to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image.

[0081] In step 101, in order to improve the robustness of the target registration parameter registration, the present invention divides the optical satellite panchromatic image according to a preset grid to obtain all optical satellite panchromatic sub-images, and divides the multispectral image according to the preset grid to obtain all multispectral sub-images. For each subgrid in the preset grid, there is an optical satellite panchromatic sub-image and a multispectral sub-image, that is, an image registration group is constituted. The present invention first determines each image registration group corresponding to all subgrids.

[0082] In step 102, for each image registration group, step 102 is performed once. Specifically, a voting parameter space is constructed according to the edge point set of the optical satellite panchromatic sub-image and the edge point set of the multispectral sub-image, and the first registration parameter corresponding to the image registration group is determined according to the voting parameter space. Since the present invention does not register the orthophotos of the panchromatic satellite image and the multispectral fusion satellite image, but registers the panchromatic satellite image and the multispectral fusion satellite image, the registration accuracy needs to be considered when considering the registration. The same-name point is a correspondence between the feature extracted from the left reference image and the feature extracted from the right target image, that is, there is a one-to-one correspondence between several elements in the left set and several elements in the right set, and there is a spatial geometric constraint between the two images to be matched according to the imaging model, and the same-name point needs to satisfy the imaging geometric constraint. The present invention uses the possible correspondence between the elements in the left and right sets to vote, and the cumulative parameter space of the vote is the parameter space composed of the imaging geometric constraint parameters between the images. The goal is to determine the first registration parameter between the satellite images, and the optimal translation amount is determined in the parameter space by the voting method.

[0083] Specifically, the present invention uses the Canny operator to extract edge points from the optical satellite panchromatic sub-image and the multispectral sub-image, respectively, and each edge point contains pixel coordinates and edge direction. The edge point is projected, and two sets are constructed, A is the edge point set in the optical satellite panchromatic sub-image, and B is the edge point set in the multispectral sub-image. The voting parameter space is established, the horizontal axis represents the registration parameter b0, and the vertical axis represents the registration parameter a0. The translation amount is calculated for each possible correspondence, and 1 is accumulated in the corresponding grid in the parameter space. Finally, the translation amount of the cell corresponding to the peak in the cumulative space is determined as the first registration parameter. By voting and accumulating in the parameter space, the best first registration parameter can be effectively found, thereby realizing the coarse matching positioning between satellite images. Obtaining the first registration parameter can help improve the accuracy and stability of the registration, and provide a reliable basis for subsequent registration and fusion operations. In order to weaken the systematic error of the rational function coefficient (RPC) parameter of the multispectral image and make the relative position relationship between the optical satellite panchromatic image and the multispectral image more accurate, the present invention introduces a voting algorithm to correct the initial predicted position and register it, so that the initial position accuracy reaches about 20 to 30 pixels.

[0084] Optionally, the present invention converts the optical satellite panchromatic sub-image and the multispectral sub-image into the frequency domain through Fourier transform, and then calculates the phase correlation between them, and determines the translation amount between the images by maximizing the phase correlation.

[0085] Specifically, a two-dimensional Fourier transform is performed on the optical satellite panchromatic sub-image and the multispectral sub-image to convert them into the frequency domain. The Fourier transform can convert the image from the spatial domain to the frequency domain and represent the image as a superposition of frequency components. Then, phase information is extracted from the spectrum after the Fourier transform. The phase information reflects the relative position relationship between components of different frequencies in the image, that is, the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image are extracted by Fourier transform. Then, the correlation between the frequency domain phases of the first phase information and the second phase information is calculated. The phase correlation can be obtained by calculating the product of the two spectrum phases, that is, the product of the two spectrum phases is inverse Fourier transformed to obtain a correlation image, and the maximum value is found in the correlation image. The position corresponding to the maximum value is the optimal translation amount, that is, the second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information.

[0086] Optionally, the position of the maximum value indicates that the phases of the optical satellite panchromatic sub-image and the multispectral sub-image are most consistent at this position, that is, the best matching position. The present invention realizes accurate image registration based on the acquired second registration parameter, which is particularly suitable for image registration scenarios requiring high precision and high stability. Its advantage also lies in its strong robustness to noise and brightness changes, and can effectively deal with some complex image registration problems.

[0087] In step 103, the present invention traverses all image registration groups to determine all first registration parameters and all second registration parameters, and determines a registration parameter sequence according to all first registration parameters and all second registration parameters, and determines a registration parameter intermediate value from the registration parameter sequence as a target registration parameter, including:

[0088] Sorting all first registration parameters and all second registration parameters in descending order to determine a registration parameter sequence;

[0089] Determining the intermediate value of the registration parameter as the target registration parameter from the registration parameter sequence comprises:

[0090] When m+n is an odd number, ;

[0091] When m+n is an even number, ;

[0092] Where m is the total number of the first registration parameters, n is the total number of the second registration parameters, is the target registration parameter, is the registration parameter sequence, .

[0093] Optionally, all first registration parameters and all second registration parameters are sorted in descending order to determine a registration parameter sequence When m+n is an odd number, the middle value is selected as the target registration parameter. When m+n is an even number, the average of the two middle parameters is taken as the target registration parameter. Since the present invention selects the parameter in the middle position as the target registration parameter among all the first registration parameters and the second registration parameters, it not only takes into account the registration accuracy, but also can effectively reduce the influence of extreme values ​​on the fusion results, thereby improving the robustness of the algorithm.

[0094] In step 104, the multispectral image is roughly registered using the target registration parameters to obtain a registered image, including:

[0095] (1)

[0096] in, is the position coordinate of the multispectral image, is the position coordinate of the image after registration, where and is the target registration parameter, , , , The preset rotation parameters.

[0097] Optionally, the present invention needs to correct row and column errors by analyzing the influence of satellite system parameters on the geometric accuracy of the image, wherein the row parameters absorb the influence of orbit and attitude in the row direction, and the column parameters absorb the influence of orbit and attitude in the column direction. For the affine transformation shown in formula (1), when there is only one registration point in the image, it can be set , only the target registration parameters need to be determined and When there are two registration points, , , that is, solving the target registration parameters and two preset rotation parameters; when there are three registration points, the target registration parameters and four preset rotation parameters can be solved; when there are more than three registration points, all preset rotation parameters can be calculated by the least squares adjustment method. After the satellite image is oriented, the two left and right images can be used to perform spatial forward intersection of the stereo image pair.

[0098] Optionally, fusing the optical satellite panchromatic image and the registered image to obtain a fused image includes:

[0099] Color transformation (Intensity, Hue, Saturation, IHS) transforms the registered image to obtain a brightness component, a hue component, and a saturation component;

[0100] Performing histogram matching between the grayscale image corresponding to the optical satellite panchromatic image and the brightness component to determine the matched brightness component;

[0101] The IHS inverse transform processes the matched brightness component, hue component and saturation component to obtain the fused image.

[0102] Alternatively, in the design of remote sensing imaging systems, spatial resolution and spectral resolution cannot be achieved at the same time. The reason is that: affected by the radiation quality characteristics of the remote sensor, the sensor's ability to sense electromagnetic waves reflected or radiated from the target object is limited, and the spectral band of the multispectral imaging system is very narrow, and the instantaneous field of view of the high spatial resolution system is also very small; the amount of data that can be collected by the sensor is limited, and the ability to transmit from the remote sensing satellite to the ground station is limited. Therefore, satellite images have adopted the method of simultaneously acquiring high spatial resolution panchromatic images and high spectral resolution multispectral images from different channels.

[0103] Therefore, remote sensing image fusion technology refers to the technology of processing the images of the same target collected by multiple source channels, extracting the information of each channel to composite the multi-source remote sensing images, and forming a unified image or comprehensively utilizing the information of each image. Its essence is to fuse the hyperspectral image of geospatial information and the panchromatic image of high spatial quality and low spectral information. Through image fusion technology, the redundant information in time, spectrum and spatial resolution can be processed with certain algorithms to obtain more accurate and richer information than any single image data, generate image data with high spatial resolution and spectral characteristics, and improve the utilization efficiency of images.

[0104] The image fusion method based on IHS change of the present invention converts the registered image into three independent components and IHS space images with small correlation through IHS change. In the fusion process, the brightness component containing the target space information in the image is replaced by the full-color image. In order to prevent the problem of a large gap between the synthesized image and the original multi-spectral image due to the inconsistent brightness range of the two, it is necessary to match the brightness of the full-color image and the brightness component so that the brightness of the two tends to be within a range and has a high correlation. The histogram matching method is an image processing method commonly used in data image processing to keep the brightness of the two images as consistent as possible. By performing histogram matching between the full-color image and the brightness component image obtained by IHS transformation, the correlation between the replacement and the replaced part is increased, and finally the distortion of the spectrum is reduced. After the component replacement is completed, the IHS inverse transformation is performed to obtain the fused image.

[0105] Optionally, in order to ensure the fusion effect of multi-source satellite images, the present invention successfully retains spectral information while increasing spatial resolution and enriching ground information, ensuring the fusion effect. The basic principle is to use the least squares rule to best match the panchromatic band and multispectral band to be fused with the input reference band, and adjust the grayscale distribution of a single band to reduce the fusion color, and use statistical methods to solve the standardization and automation problems of the fusion process, thereby obtaining a higher tone fidelity effect. Compared with other fusion algorithms, the present invention is based on the least squares algorithm to best match the grayscale values ​​of the bands involved in the fusion image, so that the grayscale values ​​between the original multispectral, panchromatic data and the fused multispectral, panchromatic data are similar. The fused image retains the color fidelity while retaining the texture details and spatial distribution details of the object; it also performs a unique statistical analysis operation on all bands involved in the operation, and eliminates the dependence of the fusion effect on the experience of technicians and the quality of its own quality through analysis, thereby improving the fusion effect.

[0106] Optionally, after fusing the optical satellite panchromatic image and the registered image to obtain a fused image, the method further includes:

[0107] Determining image features according to the spatial resolution, noise level and preset clarity of the fused image, and determining the degree of image sharpening according to the image features and preset sharpening parameters;

[0108] According to the image sharpening degree, determining a target Gaussian kernel from a correspondence between a preset sharpening degree and a preset Gaussian kernel;

[0109] Processing the fused image using the target Gaussian kernel to obtain a convolved image;

[0110] Performing pixel-by-pixel subtraction processing on the fused image and the convolved image to obtain a difference image;

[0111] Processing the difference image according to a preset sharpening strength coefficient to obtain a sharpened difference image;

[0112] The fused image and the sharpened difference image are superimposed pixel by pixel to obtain an image enhanced image.

[0113] Optionally, the present invention first needs to analyze the spatial resolution of the fused image, that is, the actual size of the ground represented by each pixel. Higher spatial resolution usually requires a smaller Gaussian kernel to retain more detail information; secondly, the noise level of the remote sensing image needs to be considered. If the image is subject to greater noise interference, a larger Gaussian kernel may need to be selected to smooth the noise and reduce the impact; at the same time, the required clarity and contrast should also be determined according to user needs. If the clarity and contrast of the image need to be improved, a smaller Gaussian kernel can be selected to retain more detail information, that is, starting from three dimensions, the image features are determined according to the spatial resolution, noise level and preset clarity of the fused image.

[0114] Optionally, the degree of image sharpening is determined based on the image features and preset sharpening parameters. The preset sharpening parameters are preset coefficients used to characterize the required sharpening effect. By controlling the preset sharpening parameters, the edge and detail information in the difference image can be enhanced to make it more suitable for superimposition on the fused image.

[0115] Optionally, the correspondence between the degree of sharpening and the Gaussian kernel is determined and stored in advance, so that a target Gaussian kernel can be determined from the correspondence between a preset degree of sharpening and a preset Gaussian kernel according to the degree of sharpening of the image. The present invention selects the size and standard deviation of the Gaussian kernel according to the degree of sharpening of the image. If it is necessary to balance the relationship between blur and detail retention, Gaussian kernels of different sizes and standard deviations can be tried to find the most suitable target Gaussian kernel.

[0116] Optionally, the present invention applies the target Gaussian kernel to each pixel of the remote sensing image, which can be achieved by a convolution operation, that is, a weighted summation of the target Gaussian kernel and each pixel in the image and its neighborhood is performed. This process smoothes the noise and details in the image and highlights the low-frequency components, thereby obtaining a blurred image. The use of the target Gaussian kernel to process the fused image to obtain the convolved image includes:

[0117] (2)

[0118] in, is any pixel value of the convolution image, is any pixel value in the neighborhood, is the size of the Gaussian kernel, is the standard deviation of the Gaussian kernel.

[0119] Optionally, the standard deviation of the Gaussian kernel determines the width of the kernel, that is, the degree of blur. A larger standard deviation will result in a greater blurring effect, while a smaller standard deviation will produce a slight blur. When selecting the standard deviation, it is necessary to adjust it according to the characteristics of the image and the desired degree of blur. The size of the Gaussian kernel determines the range of the pixel neighborhood considered in the convolution operation. Selecting a kernel of an appropriate size can ensure that the blurring operation fully smoothes the image while avoiding excessive blurring. By adjusting the standard deviation and size of the Gaussian kernel, the effect of the blurring operation can be controlled. Generally, the larger the standard deviation, the more obvious the blurring effect; the larger the kernel size, the more significant the blurring effect. The present invention can select appropriate Gaussian kernel parameters according to specific needs and image characteristics to achieve different degrees of blurring.

[0120] The fused image and the convolved image are subjected to difference processing pixel by pixel to obtain a difference image; the difference image is processed according to a preset sharpening intensity coefficient to obtain a sharpened difference image; the fused image and the sharpened difference image are superimposed pixel by pixel to obtain an image enhanced image.

[0121] Optionally, the fused image obtained by the present invention also needs to perform edge enhancement to obtain an enhanced fused image, enhance the edge features of the object, so that the edge is clearer and more convenient for stereo mapping. Specifically, the convolved image and the fused image are subtracted pixel by pixel to extract the edge and detail information that is not retained by the blurring process in the fused image. The subtraction operation obtains a difference image, which contains the difference between the convolved image and the fused image. Then, according to the characteristics of the remote sensing image and the required sharpening effect, an appropriate preset sharpening intensity coefficient is selected. By controlling the sharpening degree, the edge and detail information in the difference image can be enhanced to obtain a sharpened difference image, which is more suitable for superimposing on the original image. Finally, the sharpened difference image is superimposed pixel by pixel on the fused image to obtain an image enhanced image. The image enhanced image will contain the information of the fused image and the enhanced edge and detail information, thereby improving the clarity and contrast of the fused image.

[0122] Those skilled in the art understand that when using panchromatic images for manual stereo mapping, it is difficult to accurately capture the details of the objects, which affects the accuracy and production efficiency of stereo mapping. Current stereo mapping software does not support the automatic fusion of 0-2 level optical satellite panchromatic and multispectral images. Therefore, in the production practice of satellite stereo mapping, grayscale panchromatic satellite base maps are still used for manual operations. Although the panchromatic fusion algorithm has existed for a long time, related research has mainly focused on the orthophoto product generation stage, and has not fully considered the panchromatic multispectral fusion problem of 0-2 level satellite image products. The present invention makes full use of the potential advantages of optical satellite panchromatic images and multispectral images, solves the difficulties in the production practice of optical satellite image stereo mapping, improves its accuracy and production efficiency, promotes the application and development of satellite stereo mapping, and provides strong support for technological innovation and progress in related fields.

[0123] Optionally, the present invention provides reliable data support for high-precision stereo mapping by deeply studying the fusion algorithm of panchromatic images and multispectral images and the preprocessing of stereo mapping after fusion, and provides more accurate and efficient data support for surveying and mapping work by accurately extracting and expressing the surface morphology. The innovation of satellite sensor technology improves the spatial resolution of images and more accurately captures surface features and details. The high-precision processing capabilities of stereo mapping technology in image matching, feature extraction and model reconstruction are enhanced, and the mapping accuracy and detail presentation are improved. The development trend of refined stereo mapping is not only of great significance in the field of geographic information, but will also have a profound impact on related fields of geographic information.

[0124] Figure 2 This is a second flow chart of the registration and fusion method of the optical satellite panchromatic image and the multispectral image provided by the present invention, wherein the multispectral image is roughly registered using the target registration parameters to obtain the registered image, including:

[0125] Step 201, using the target registration parameters to perform coarse registration on the multispectral image to obtain a coarse registered image;

[0126] Step 202: extract matching feature points from the coarsely registered image, intersect the matching feature points with a preset digital elevation model through an image-to-ground forward solution algorithm, and determine the position coordinates of the feature points;

[0127] Step 203, using a preset rational function model to calculate the position coordinates of the feature points, and solving the target coordinate positions of the matching feature points in the optical satellite panchromatic image;

[0128] Step 204: update the coarsely registered image according to the feature point position coordinates to obtain the registered image, and update the optical satellite panchromatic image according to the target coordinate position, so as to use the updated optical satellite panchromatic image and the registered image to perform pyramid image matching step by step to obtain a fused image.

[0129] In step 201, the present invention provides another method for determining the registered image. Different from directly using the coarse registered image for subsequent image fusion, the coarse registered image determined in step 201 will also be used for subsequent precise matching and registration.

[0130] In step 202, extracting matching feature points from the coarsely registered image includes:

[0131] For each pixel in the coarse registration image, the sum of squares of grayscale differences of adjacent pixels in each direction in the first preset window corresponding to the pixel is calculated, and the value with the minimum sum of squares is determined as the characteristic value of the pixel;

[0132] Traverse all pixel points and determine all pixel points with feature values ​​greater than a preset threshold as initial candidate points;

[0133] For each initial candidate point, a grayscale correlation coefficient corresponding to the initial candidate point is calculated according to the grayscale correlation between the initial candidate point and each pixel point in the first preset window, and initial candidate points whose grayscale correlation coefficients are less than a preset coefficient are eliminated to obtain intermediate candidate points;

[0134] Dividing the coarse registration image into a plurality of second preset windows, and for each second preset window, determining the middle candidate point with the largest eigenvalue in the second preset window as the target candidate point;

[0135] For each target candidate point, a third preset window is constructed with the target candidate point as the center, and the horizontal and vertical gradient values ​​of each pixel point in the third preset window are calculated to construct a gradient matrix; the gradient matrix is ​​processed by eigenvalue decomposition to obtain a first eigenvalue and a second eigenvalue, and a target candidate point whose target eigenvalue is greater than a preset eigenvalue is determined as the matching feature point;

[0136] The target eigenvalue is the minimum value of the first eigenvalue and the second eigenvalue.

[0137] Optionally, as the first feature point screening: for each pixel point in the coarse registration image, the present invention calculates the sum of the squares of the grayscale differences of neighboring pixels in four directions, namely up, down, left and right, in the first preset window centered on any pixel point, and takes the minimum sum of the squares of the grayscale differences in the four directions as the feature value of the pixel point. Further, a preset threshold is set, and the pixel points with feature values ​​greater than the preset threshold are taken as initial candidate points. Optionally, the selection of the preset threshold should ensure that the required feature points are included in the initial candidate points, while avoiding the inclusion of too many non-feature points.

[0138] Optionally, for each initial candidate point, calculating a grayscale correlation coefficient corresponding to the initial candidate point according to a grayscale correlation between the initial candidate point and each pixel point in the first preset window includes:

[0139] (3)

[0140] in, is the grayscale correlation coefficient corresponding to any initial candidate point, is the grayscale of any pixel in the first preset window, is the grayscale mean value in the first preset window, for The derivative of , k is the number of rows of the first preset window, and l is the number of columns of the first preset window.

[0141] Optionally, the matching accuracy of the grayscale correlation coefficient is related to the texture structure in the image window. The richer the grayscale information in the image window, the higher the matching accuracy; on the contrary, if the grayscale in the image window does not change (such as water surface, snow, etc.), image matching cannot be performed. Therefore, it is necessary to extract feature points before matching to avoid matching in areas where the grayscale changes are not obvious or have not changed at all, so as to improve the matching accuracy and reliability. In addition, the matching accuracy of the grayscale correlation coefficient is related to the signal-to-noise ratio of the image window, and the influence of noise can be reduced by increasing the matching window.

[0142] Optionally, by comparing an image window as the target area on the left image with an image window of the same size corresponding to the search area on the right image, the correlation coefficient of all search areas is obtained, wherein the coordinates of the center point of the center pixel of the search area window where the maximum correlation coefficient is located are considered to be the conjugate point sought, i.e., the same name point. Due to the difference in sampling of the left and right images, the center point of the same name pixel is generally not the real same name point. The real same name point may deviate from the center point of the pixel within half a pixel, which causes the correlation to produce errors.

[0143] Further, as the second feature point screening, a preset coefficient is set, and the initial candidate points with grayscale correlation coefficients less than the preset coefficient are eliminated to obtain intermediate candidate points. It can be seen from the spectral analysis of the correlation function that when there are fewer high-frequency components in the signal, the correlation function curve is relatively gentle, but the relevant pull-in range is larger; conversely, when there are more high-frequency components, the correlation function curve is steeper, the correlation accuracy is higher, but the relevant pull-in range is smaller. In addition, when there is high-frequency narrowband random noise in the signal or there is a strong high-frequency signal in the signal, the correlation function has multiple peaks, so there will be wrong matching. Compare the grayscale correlation coefficient with the preset coefficient, and remove the points with grayscale correlation coefficients less than the preset coefficient as matching error points. Multiple peaks may appear in the search window, so that the point with the maximum grayscale correlation coefficient may not be a true point of the same name. The present invention compares the grayscale correlation coefficient of each initial candidate point with the preset coefficient. If the grayscale correlation coefficient of a certain initial candidate point is less than the preset coefficient, the initial candidate point is treated as a possible wrong matching point pair and is eliminated, thereby ensuring the matching accuracy and reliability of the image.

[0144] Furthermore, as the third feature point screening, the present invention divides the coarsely registered image into multiple second preset windows, performs a screening on each second preset window, determines the middle candidate point with the largest eigenvalue in the second preset window as the target candidate point, retains the middle candidate point with the largest eigenvalue in each second preset window, removes other middle candidate points, and the retained middle candidate point has the largest grayscale change in the second preset window.

[0145] Further, as the fourth feature point screening, for each target candidate point, a third preset window or neighborhood is constructed with the target candidate point as the center, and the horizontal and vertical gradient values ​​of each pixel point in the third preset window are calculated. The Sobel operator or other gradient operators can be used to calculate the gradients in the horizontal and vertical directions, and a gradient matrix is ​​constructed according to the calculated horizontal and vertical gradient values; the gradient matrix is ​​processed by eigenvalue decomposition to obtain the first eigenvalue and the second eigenvalue, and the target candidate point whose target eigenvalue is greater than the preset eigenvalue is determined as the matching feature point; the target eigenvalue is the minimum value of the first eigenvalue and the second eigenvalue, indicating that there is a significant grayscale change around the matching feature point. The present invention selects the minimum eigenvalue as the target eigenvalue, and selects a larger target candidate point in the target eigenvalue as the matching feature point, which can effectively extract feature points with significant grayscale changes and further improve the accuracy and robustness of image registration.

[0146] Further, after determining the matching feature points, the present invention intersects the matching feature points with a preset digital elevation model through an image-ground positive solution algorithm to determine the position coordinates of the feature points. The preset rational function model is a mathematical model used to describe the geometric relationship between a digital image and the ground, which includes internal orientation elements, external orientation elements, and radial distortion parameters in photogrammetry. The preset rational function model can accurately describe the projection relationship between image pixels and ground coordinates, and is the basis for calculating orientation parameters. The present invention first determines the internal orientation elements and external orientation elements, and then presets a digital elevation model, including elevation information of each point on the ground. The elevation information can come from a laser radar, measurement data, or other digital elevation models. Then, the preset rational function model and external orientation elements are used to convert the pixel coordinates of the matching feature points into the ground coordinates of the matching feature points. This step needs to consider information such as the internal orientation elements, the external orientation elements, and the coordinates of the ground control points. Then, the ground coordinates of the matching feature points are intersected with the digital elevation model to determine the position coordinates of the matching feature points on the ground. This step needs to consider factors such as the resolution and accuracy of the digital elevation model.

[0147] In step 203, the prerequisite for pixel-level fusion is to achieve high-precision registration between the panchromatic band and the multispectral band of the satellite image. The registration between the panchromatic and multispectral bands of the traditional satellite image is usually performed by polynomial fitting between the two images using matching points of the same name. Since simple polynomial fitting cannot reflect the real imaging model of the satellite image, it cannot reliably eliminate the gross errors of matching. The present invention can solve the target coordinate position of the feature point in the panchromatic image through directional parameter calculation, and the target coordinate position can help determine the position and coordinates of the matching feature point in the image.

[0148] In step 204, the present invention comprehensively considers the correctness and accuracy of the correlation results, and implements a correlation strategy from coarse to fine in image registration, first performs initial correlation through low-pass filtering, finds the rough position of the same-name point, and then uses high-frequency information for precise correlation. The pyramid image step-by-step matching strategy of the present invention is to extract matching feature points from the coarse registration image, and then predict the satellite image according to the preset rational function model. After matching all coarse registration images covered by the satellite image on each level of the pyramid image, the matching feature points are used to solve the orientation parameter value of the satellite image through the preset rational function model orientation method. During the orientation process, gross error detection and automatic elimination are performed simultaneously to ensure the reliability of solving the orientation parameters. Through step-by-step pyramid image matching, the orientation parameters and control point results with gradually improved accuracy are obtained.

[0149] Optionally, the present invention can also verify and evaluate the accuracy of the results of the orientation parameter calculation to ensure that the deviation between the full-color image coordinates of the matching feature points and the ground truth position is within an acceptable range.

[0150] The present invention provides a high-precision registration technology for optical satellite panchromatic images and multispectral images, achieving accurate alignment of the two images, and providing a reliable data basis for subsequent panchromatic fusion and stereo mapping. The present invention also provides a panchromatic fusion technology for 1-2 level optical satellite panchromatic images and multispectral images, so as to make full use of the high resolution of panchromatic images and the color information of multispectral images, generate fused images, maintain high resolution and have rich color information, and improve the accuracy and visualization effect of stereo mapping.

[0151] Figure 3 It is a structural schematic diagram of the optical satellite panchromatic image and multispectral image registration and fusion device provided by the present invention, the optical satellite panchromatic image and multispectral image registration and fusion device comprises a segmentation unit 1, which is used to segment the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group comprises an optical satellite panchromatic sub-image and a multispectral sub-image, the working principle of the segmentation unit 1 can refer to the aforementioned step 101, which will not be repeated here.

[0152] The optical satellite panchromatic image and multispectral image registration and fusion device also includes a determination unit 2, which is used to construct a voting parameter space for each image registration group according to the edge point set of the optical satellite panchromatic sub-image and the edge point set of the multispectral sub-image, and determine the first registration parameter corresponding to the image registration group according to the voting parameter space; use Fourier transform to extract the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image, and determine the second registration parameter corresponding to the correlation image according to the first phase information and the second phase information. The working principle of the determination unit 2 can refer to the aforementioned step 102 and will not be repeated here.

[0153] The optical satellite panchromatic image and multispectral image registration and fusion device also includes a traversal unit 3, which is used to traverse all image registration groups, determine a registration parameter sequence according to all first registration parameters and all second registration parameters, and determine the intermediate value of the registration parameter from the registration parameter sequence as the target registration parameter. The working principle of the traversal unit 3 can be referred to the aforementioned step 103 and will not be repeated here.

[0154] The optical satellite panchromatic image and multispectral image registration and fusion device also includes a fusion unit 4, which is used to use the target registration parameters to roughly register the multispectral image to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image. The working principle of the fusion unit 4 can refer to the aforementioned step 104 and will not be repeated here.

[0155] The present invention divides the optical satellite panchromatic image and the multispectral image into a plurality of image registration groups, and uses two registration parameter determination methods to determine the target registration parameters used for registration, thereby improving the registration accuracy of the image registration, and then realizing the registration fusion with high geometric accuracy in stereo mapping. Since the target registration parameter is the intermediate value of the registration parameter, the influence of the extreme value on the fusion result is reduced, and the robustness of the algorithm to abnormal values ​​is enhanced, making the final image registration more stable and reliable. The present invention also uses a multi-level feature screening method to extract matching feature points from the coarse registration image, greatly improving the accuracy, reliability and robustness of the matching feature point selection, and finally, by performing image enhancement on the fused image, the concept of image sharpening is introduced, and the target Gaussian kernel is more accurately determined, thereby improving the clarity and contrast of the fused image.

[0156] Figure 4 Schematic diagram of the structure of the electronic device provided by the present invention. Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communications bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the registration and fusion method of the optical satellite panchromatic image and the multispectral image, the method comprising: dividing the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group comprising an optical satellite panchromatic sub-image and a multispectral sub-image; for each image registration group, constructing a voting parameter space according to the edge point set of the optical satellite panchromatic sub-image and the edge point set of the multispectral sub-image, and determining the first registration parameter corresponding to the image registration group according to the voting parameter space; using Fourier transform to obtain the first registration parameter of the image registration group; The leaf transform extracts the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image, and determines the second registration parameter corresponding to the correlation image according to the first phase information and the second phase information; traverses all image registration groups, determines the registration parameter sequence according to all the first registration parameters and all the second registration parameters, and determines the intermediate value of the registration parameter from the registration parameter sequence as the target registration parameter; uses the target registration parameter to perform coarse registration on the multispectral image to obtain a registered image, and fuses the optical satellite panchromatic image and the registered image to obtain a fused image.

[0157] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a registration and fusion method for an optical satellite panchromatic image and a multispectral image provided by the above methods. The method includes: dividing the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image; for each image registration group, constructing a voting parameter space according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image , determine the first registration parameter corresponding to the image registration group according to the voting parameter space; use Fourier transform to extract the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image, and determine the second registration parameter corresponding to the correlation image according to the first phase information and the second phase information; traverse all image registration groups, determine the registration parameter sequence according to all the first registration parameters and all the second registration parameters, and determine the intermediate value of the registration parameter from the registration parameter sequence as the target registration parameter; use the target registration parameter to roughly register the multispectral image to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image.

[0159] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the registration and fusion methods of optical satellite panchromatic images and multispectral images provided by the above methods, the method comprising: dividing the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group comprising an optical satellite panchromatic sub-image and a multispectral sub-image; for each image registration group, constructing a voting parameter space according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image, and determining the image registration group according to the voting parameter space. The first registration parameter corresponding to the image registration group is obtained; the first phase information of the optical satellite panchromatic sub-image and the second phase information of the multispectral sub-image are extracted by Fourier transform, and the second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information; all image registration groups are traversed, and a registration parameter sequence is determined according to all the first registration parameters and all the second registration parameters, and the intermediate value of the registration parameter is determined from the registration parameter sequence as the target registration parameter; the multispectral image is roughly registered by using the target registration parameter to obtain a registered image, and the optical satellite panchromatic image and the registered image are fused to obtain a fused image.

[0160] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A registration and fusion method for optical satellite panchromatic images and multispectral images, characterized in that: include: Segment the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image; For each image registration group, a voting parameter space is constructed according to an edge point set of an optical satellite panchromatic sub-image and an edge point set of a multispectral sub-image, and a first registration parameter corresponding to the image registration group is determined according to the voting parameter space; first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image are extracted by Fourier transform, and a second registration parameter corresponding to the correlation image is determined according to the first phase information and the second phase information; Traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining an intermediate value of the registration parameter from the registration parameter sequence as a target registration parameter; Using the target registration parameters to roughly register the multispectral image to obtain a registered image, and fusing the optical satellite panchromatic image and the registered image to obtain a fused image; The method of using the target registration parameters to roughly register the multispectral image to obtain a registered image includes: Using the target registration parameters to roughly register the multispectral image to obtain a roughly registered image; Extracting matching feature points from the coarsely registered image, intersecting the matching feature points with a preset digital elevation model through an image-to-ground forward solution algorithm, and determining the position coordinates of the feature points; The position coordinates of the feature points are calculated by using a preset rational function model to solve the target coordinate positions of the matching feature points in the optical satellite panchromatic image; The coarsely registered image is updated according to the position coordinates of the feature points to obtain the registered image, and the optical satellite panchromatic image is updated according to the target coordinate position, so as to use the updated optical satellite panchromatic image and the registered image to perform pyramid image matching step by step to obtain a fused image.

2. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 1, characterized in that: The step of determining a registration parameter sequence according to all the first registration parameters and all the second registration parameters, and determining an intermediate value of the registration parameter from the registration parameter sequence as a target registration parameter comprises: Sorting all first registration parameters and all second registration parameters in descending order to determine a registration parameter sequence; Determining the intermediate value of the registration parameter as the target registration parameter from the registration parameter sequence comprises: When m+n is an odd number, ; When m+n is an even number, ; Where m is the total number of the first registration parameters, n is the total number of the second registration parameters, is the target registration parameter, is the registration parameter sequence, .

3. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 1, characterized in that: The method of using the target registration parameters to roughly register the multispectral image to obtain a registered image includes: ; in, is the position coordinate of the multispectral image, is the position coordinate of the image after registration, where and is the target registration parameter, , , , The preset rotation parameters.

4. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 1, characterized in that: The step of extracting matching feature points from the coarsely registered image comprises: For each pixel in the coarse registration image, the sum of squares of grayscale differences of adjacent pixels in each direction in the first preset window corresponding to the pixel is calculated, and the value with the minimum sum of squares is determined as the characteristic value of the pixel; Traverse all pixel points and determine all pixel points with feature values ​​greater than a preset threshold as initial candidate points; For each initial candidate point, a grayscale correlation coefficient corresponding to the initial candidate point is calculated according to the grayscale correlation between the initial candidate point and each pixel point in the first preset window, and initial candidate points whose grayscale correlation coefficients are less than a preset coefficient are eliminated to obtain intermediate candidate points; Dividing the coarse registration image into a plurality of second preset windows, and for each second preset window, determining the middle candidate point with the largest eigenvalue in the second preset window as the target candidate point; For each target candidate point, a third preset window is constructed with the target candidate point as the center, and the horizontal and vertical gradient values ​​of each pixel point in the third preset window are calculated to construct a gradient matrix; the gradient matrix is ​​processed by eigenvalue decomposition to obtain a first eigenvalue and a second eigenvalue, and a target candidate point whose target eigenvalue is greater than a preset eigenvalue is determined as the matching feature point; The target eigenvalue is the minimum value of the first eigenvalue and the second eigenvalue.

5. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 4, characterized in that: The calculating the grayscale correlation coefficient corresponding to the initial candidate point according to the grayscale correlation between the initial candidate point and each pixel point in the first preset window includes: ; in, is the grayscale correlation coefficient corresponding to any initial candidate point, is the grayscale of any pixel in the first preset window, is the grayscale mean value in the first preset window, for The derivative of , k is the number of rows of the first preset window, and l is the number of columns of the first preset window.

6. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 1, characterized in that: The fusing the optical satellite panchromatic image and the registered image to obtain a fused image comprises: The registered image is processed by IHS transformation to obtain a brightness component, a hue component and a saturation component; Performing histogram matching between the grayscale image corresponding to the optical satellite panchromatic image and the brightness component to determine the matched brightness component; The IHS inverse transform processes the matched brightness component, hue component and saturation component to obtain the fused image.

7. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 1, characterized in that: After fusing the optical satellite panchromatic image and the registered image to obtain a fused image, the method further includes: Determining image features according to the spatial resolution, noise level and preset clarity of the fused image, and determining the degree of image sharpening according to the image features and preset sharpening parameters; According to the image sharpening degree, determining a target Gaussian kernel from a correspondence between a preset sharpening degree and a preset Gaussian kernel; Processing the fused image using the target Gaussian kernel to obtain a convolved image; Performing pixel-by-pixel subtraction processing on the fused image and the convolved image to obtain a difference image; Processing the difference image according to a preset sharpening strength coefficient to obtain a sharpened difference image; The fused image and the sharpened difference image are superimposed pixel by pixel to obtain an image enhanced image.

8. The registration and fusion method of optical satellite panchromatic image and multispectral image according to claim 7, characterized in that: The step of processing the fused image using the target Gaussian kernel to obtain a convolved image includes: ; in, is any pixel value of the convolution image, is any pixel value in the neighborhood, is the size of the Gaussian kernel, is the standard deviation of the Gaussian kernel.

9. A registration and fusion device for optical satellite panchromatic images and multispectral images, characterized in that: include: A segmentation unit, configured to segment the optical satellite panchromatic image and the multispectral image according to a preset grid to obtain all image registration groups, each image registration group including an optical satellite panchromatic sub-image and a multispectral sub-image; A determination unit is used to construct a voting parameter space for each image registration group according to an edge point set of the optical satellite panchromatic sub-image and an edge point set of the multispectral sub-image, and determine a first registration parameter corresponding to the image registration group according to the voting parameter space; extract first phase information of the optical satellite panchromatic sub-image and second phase information of the multispectral sub-image by Fourier transform, and determine a second registration parameter corresponding to the correlation image according to the first phase information and the second phase information; A traversal unit, used for traversing all image registration groups, determining a registration parameter sequence according to all first registration parameters and all second registration parameters, and determining a registration parameter intermediate value from the registration parameter sequence as a target registration parameter; A fusion unit, configured to perform coarse registration on the multispectral image using the target registration parameters to obtain a registered image, and fuse the optical satellite panchromatic image and the registered image to obtain a fused image; The method of using the target registration parameters to roughly register the multispectral image to obtain a registered image includes: Using the target registration parameters to coarsely register the multispectral image to obtain a coarsely registered image; Extracting matching feature points from the coarsely registered image, intersecting the matching feature points with a preset digital elevation model through an image-to-ground forward solution algorithm, and determining the position coordinates of the feature points; The position coordinates of the feature points are calculated by using a preset rational function model to solve the target coordinate positions of the matching feature points in the optical satellite panchromatic image; The coarsely registered image is updated according to the position coordinates of the feature points to obtain the registered image, and the optical satellite panchromatic image is updated according to the target coordinate position, so as to use the updated optical satellite panchromatic image and the registered image to perform pyramid image matching step by step to obtain a fused image.