A Hyperspectral Image Stitching Method and System Based on Spectral Fitting

Abstract

This invention discloses a hyperspectral image stitching method and system based on spectral fitting. The method includes extracting feature points from each hyperspectral image, matching feature points from multiple hyperspectral images, transforming feature points obtained from the feature points, constructing a spectral fitting function for the overlapping region based on the set of feature point pairs, obtaining the corresponding spectral coefficients by minimizing the spectral fitting function, calculating the new spectral values ​​of pixels in the overlapping region, keeping the non-overlapping region unchanged, and then fusing them to obtain the final panoramic image, achieving a better stitching effect.

Description

Technical Field

[0001] This invention relates to the field of intelligent processing of remote sensing images, and in particular to a hyperspectral image stitching method and system based on spectral fitting. Background Technology

[0002] Hyperspectral remote sensing technology utilizes numerous narrow electromagnetic bands to capture detailed information about target objects. It combines the spectral and spatial characteristics of the target, achieving the fusion of image and spectral information, possessing both two-dimensional spatial and one-dimensional spectral information. Many substances are difficult to detect using traditional broadband remote sensing, but are much easier to detect using hyperspectral remote sensing. Therefore, hyperspectral remote sensing has enormous research value and development potential. Hyperspectral images possess spectral characteristics, meaning that different ground objects exhibit varying reflection and absorption characteristics for specific wavelengths of electromagnetic waves, all of which are encompassed by hyperspectral images. Simultaneously, hyperspectral images also exhibit inter-spectral correlation; that is, at the same spatial coordinates, different bands of a hyperspectral image show a certain degree of similarity because the reflectivity of the same ground object is often very close in different bands. Furthermore, multiple identical pixels in different bands still represent the same object, thus possessing the same spatial topology. Image stitching technology mainly consists of four steps: image acquisition, image preprocessing, image registration, and image fusion. During image fusion, errors in interpolation methods and fusion strategies can lead to blurring in overlapping areas, making it impossible to accurately recover the original image information. In aerial image acquisition, even with short time intervals between adjacent images, changes in weather, lighting, and shadows can cause differences in color and brightness, resulting in spectral differences. Directly registering such images can produce noticeable seams or abrupt transitions in overlapping areas. While some preprocessing methods can be used to make the grayscale levels between images more consistent, this is a relatively coarse and widespread adjustment. In panoramic images, each pixel in the overlapping area is actually a matching point pair from at least two sub-images. In hyperspectral remote sensing images, each pixel has a fixed spectral value in each band; however, there are still differences in spectral values ​​between matching feature point pairs. To achieve seamless hyperspectral image stitching, it is necessary not only to align the pixels in the overlapping area spatially but also to minimize the spectral differences of the same pixel in different bands. Therefore, it is necessary to develop new algorithms specifically designed to address issues such as stitching seams and brightness differences, taking into account the characteristics of the images.

[0003] Hyperspectral images possess both spatial and spectral characteristics. This paper proposes a method that combines these two characteristics, effectively utilizing spatial information while incorporating spectral information to assist in image stitching. This method is highly valuable for both practical application and research. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, one object of the present invention is to provide a hyperspectral image stitching method based on spectral fitting.

[0005] Another objective of this invention is to provide a hyperspectral image stitching system based on spectral fitting.

[0006] This invention proposes a hyperspectral image stitching method based on spectral fitting. Taking the set of matching point pairs in the overlapping region as a benchmark, and addressing the issue that the spectral values ​​of the same ground object may differ in the same wavelength band in reality, the method aims to minimize the spectral differences of the same pixel in different bands. Based on the idea of ​​minimizing spectral differences, image fusion is performed by constructing a spectral fitting function for the overlapping region using a strategy of minimizing the spectral differences of matching point pairs in the overlapping region. The parameter values ​​are calculated by minimizing the spectral error function, thereby obtaining the new spectral value of each pixel in the overlapping region, i.e., the fitted spectral value, completing the fusion step and effectively improving the stitching effect.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A hyperspectral image stitching method based on spectral fitting includes:

[0009] Read multiple hyperspectral images to be stitched together, and extract feature points from each hyperspectral image separately;

[0010] Feature points in multiple hyperspectral images are matched to obtain feature matching point pairs. Based on the matching relationship between images and the feature intra-points obtained after filtering, a set of feature intra-points between multiple images is obtained.

[0011] Calculate the transformation matrix between pairwise matched images;

[0012] Select the image with the most matching images as the reference image, and the remaining images as the target images;

[0013] The transformation matrix of the target image that does not directly overlap with the reference image is transformed into the transformation matrix in the coordinate system of the reference image by concatenation, while the transformation matrix of the target image that directly overlaps with the reference image remains unchanged.

[0014] Iteratively optimize the transformation matrix parameters of the image and calculate the final transformation matrix;

[0015] Based on the final transformation matrix of the image, the relative positions of the images on the same plane are obtained;

[0016] Construct a spectral fitting function for the overlapping region based on the set of feature in-point pairs;

[0017] The spectral coefficients are obtained by minimizing the spectral error function;

[0018] The obtained spectral coefficients are used to calculate the new spectral values ​​of pixels in the overlapping region, while the non-overlapping region remains unchanged, resulting in the final panoramic image.

[0019] Furthermore, based on the set of feature in-place point pairs, a spectral fitting function for the overlapping region is constructed, specifically as follows:

[0020] E(x,y)=a×m(x i ,y i )+b×m(x j ,y j )

[0021] Where m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j ) represents the original spectral value at pixel (x,y) in Figure j, and a and b are the spectral coefficients to be solved.

[0022] Furthermore, the spectral coefficients are obtained by minimizing the spectral error function, specifically:

[0023]

[0024] Where M is the total number of matching point pairs, Δ k This represents the feature in-point pair (x) corresponding to the k-th matching pair. i ,y i ) and (x j ,y j The difference in spectral values, i.e.

[0025]

[0026] In the above formula and These represent the weighted spectral values ​​of the k-th feature pair, respectively. Their form is consistent with the spectral fitting function. By inputting the spectral values ​​of all feature pairs, the values ​​of the spectral coefficients a and b can be obtained.

[0027] Furthermore, the obtained spectral coefficients are used to calculate the new spectral values ​​of pixels in the overlapping regions, while the non-overlapping regions remain unchanged, resulting in the final panoramic image, specifically:

[0028]

[0029] In the above formula, m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j) represents the original spectral value at pixel (x,y) in image j, and m′(x,y) represents the spectral value at pixel (x,y) in the final panoramic image.

[0030] Furthermore, the iterative optimization of the image transformation matrix parameters, and the calculation of the final transformation matrix, are as follows:

[0031] The error between two matched images i and j is defined as the sum of the distances between the new coordinates of all feature points after transformation by the homography transformation matrix and the original coordinates. The calculation method is as follows:

[0032]

[0033] in, and Let k be the in-feature point of the feature in images i and j, respectively. H represents all feature interior points of images i and j. ij Let represent the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between each image and its matching image, calculated as follows:

[0034]

[0035] Where n represents the number of images to be stitched, I(i) represents all images that match image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix. ij .

[0036] Furthermore, the matching relationship between images specifically involves determining whether image pairs match. The determination process is as follows:

[0037] Let the total number of interior points of the feature be n. f The number of feature in-point pairs is n i If n i >8+0.3·n f If so, then the two images are matched.

[0038] Furthermore, the RANSAC algorithm is used to filter the feature points.

[0039] Furthermore, feature points in multiple hyperspectral images are matched to obtain feature interior points between multiple images. Specifically, the ratio of nearest Euclidean distance to second nearest Euclidean distance is used to achieve coarse matching of feature descriptor vectors.

[0040] A system for implementing the hyperspectral image stitching method based on spectral fitting includes:

[0041] Image acquisition module: Reads multiple hyperspectral images to be stitched together and extracts feature points from each hyperspectral image;

[0042] Image matching module: Matches feature points in multiple hyperspectral images to obtain a set of feature points within the images;

[0043] Image registration module: Calculates the transformation matrix between pairwise matching images, and transforms the transformation matrix of the target image that does not overlap with the reference image into the transformation matrix in the coordinate system of the reference image by concatenation, while the transformation matrix of the target image that directly overlaps with the reference image remains unchanged; iteratively optimizes the transformation matrix parameters of the image, calculates the final transformation matrix, and obtains the relative position of the images on the same plane.

[0044] Image fusion module: Based on the set of feature point pairs, construct a spectral fitting function for the overlapping region; obtain spectral coefficients by minimizing the spectral error function; use the obtained spectral coefficients to calculate the new spectral values ​​of pixels in the overlapping region, while keeping the non-overlapping region unchanged, to obtain the final panoramic image.

[0045] A storage medium storing a program that, when executed by a processor, implements the hyperspectral image stitching method based on spectral fitting.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] This invention proposes a method that utilizes the spectral differences and consistency between intra-feature points. It employs a strategy of minimizing the spectral differences between matching intra-feature point pairs in overlapping regions to construct a spectral fitting function for the overlapping region. By minimizing this function, the spectral coefficients can be calculated, thereby obtaining the new spectral values ​​for each pixel in the overlapping region, resulting in a better fusion effect. Compared to traditional image stitching methods, the stitching effect using the spectral fitting fusion method is significantly improved. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the hyperspectral image stitching method based on spectral fitting of the present invention;

[0049] Figure 2(a) shows the splicing effect using the traditional method, and Figure 2(b) shows the splicing effect using the proposed method.

[0050] Figure 3 This is a flowchart of the process of the present invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0052] Example

[0053] like Figures 1-3 As shown, a hyperspectral image stitching method based on spectral fitting includes the following steps:

[0054] S1 reads multiple hyperspectral images to be stitched together and extracts feature points from each hyperspectral image;

[0055] S2 matches feature points in multiple hyperspectral images to obtain a set of feature matching pairs between multiple images;

[0056] Specifically:

[0057] A coarse matching of feature descriptor vectors is achieved using the ratio of nearest Euclidean distance to second nearest Euclidean distance. A feature point in the reference image is selected, and then the two feature points in the target image that are closest and second closest to that feature point in terms of Euclidean distance are found. These two Euclidean distance values ​​are then divided. If the result is less than a set threshold, the pair of feature points with the closest Euclidean distance is considered a match.

[0058] Let the feature point descriptor R of the reference image and the feature point descriptor S of the target image be represented as follows:

[0059] R = [r1, r2, ..., r 128 ] T

[0060] S = [s1, s2, ..., s 128 ] T

[0061] The formula for calculating the Euclidean distance between the two is as follows:

[0062]

[0063] After obtaining the nearest and second nearest Euclidean distances using the above formula, the ratio of the two distances can be used to determine whether the pair of feature points match.

[0064]

[0065] Here, threshold represents the threshold value, and its size determines the number of feature matching pairs obtained.

[0066] The reference image in this step is local (valid only in the feature matching stage). It refers to the relationship between a pair (two images) of images selected for matching each time during the feature point matching process. One image serves as the reference image for feature point matching, while the other serves as the target image for feature point matching. The relationship between them is only valid between this pair of images within this step, and the relationship between this pair of images can be interchanged (the same image can be both the target image and the reference image) without affecting the matching result.

[0067] S3 filters the obtained feature matching pairs to obtain intra-feature point pairs between multiple image pairs. The RANSAC algorithm is used to remove mismatched point pairs and retain the correctly matched point pairs, which are the intra-feature point pairs.

[0068] S4 determines whether corresponding image pairs match based on the number of feature point pairs. The specific steps are as follows: Let the total number of feature point pairs be n. f The number of feature in-point pairs obtained after filtering using the RANSAC algorithm is n. i If n i >8+0.3·n f If so, then the two images are matched.

[0069] S5 calculates the transformation matrix between pairwise matched images based on the matching relationship of the matched images and the filtered feature inlier pairs. The specific steps are as follows: Let one pair of inlier points in the two images be [x1y1]. T and [x2y2] T Transform both to [x1y11] in a homogeneous coordinate system. T and [x2y21] T After coordinates, assume the relationship between the two satisfies the homography transformation matrix H:

[0070]

[0071] By inputting the coordinates of all feature pairs, the homography transformation matrix H can be calculated.

[0072] S6 selects the image with the most matching images as the reference image, and the remaining images as the target images.

[0073] The reference image in this step is holistic and is effective throughout the entire registration process. It is selected from all the images to be stitched. Generally, there is only one reference image, which serves as the reference coordinate system for the transformation and therefore does not need to be transformed. The other images to be stitched, which are not reference images, are target images that need to be transformed.

[0074] S7 transforms the transformation matrix of the target image that does not directly overlap with the reference image into the transformation matrix in the coordinate system of the reference image by concatenation, while the transformation matrix of the target image that directly overlaps with the reference image remains unchanged.

[0075] To further explain, generally speaking, if two images have more than four matching pairs of feature points, they are considered to have overlapping regions; otherwise, the images are considered not to overlap or the overlapping area is too small to be calculated. The set of feature point pairs always lies within the overlapping regions of the images.

[0076] Feature extraction algorithms are designed for a single image, resulting in feature points that are evenly distributed across each image (including feature points in both overlapping and non-overlapping regions). Feature matching algorithms, on the other hand, are designed for the relationship between two images. They use the feature points of the two images as input and determine whether there is a common matching relationship (i.e., a transformation matrix) between these feature points. If a matching relationship exists, the feature points are considered interior points; otherwise, they are considered exterior points. Interior points are all located in overlapping regions because generally only feature points in overlapping regions can be matched.

[0077] The term "cascading" refers to the fact that the transformation relationship between two images needs to be calculated using the transformation moments of other images that overlap with each of the two images.

[0078] The S8 iterative optimization optimizes the transformation matrix parameters of the image and calculates the final transformation matrix. The specific steps are as follows: The error between two matching images i and j is defined as the sum of the distances between the new coordinate points of all feature points after the homography transformation matrix and the original coordinate points. The calculation method is as follows:

[0079]

[0080] in, and Let k be the in-feature point of the feature in images i and j, respectively. H represents all feature interior points of images i and j. ij Let represent the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between each image and its matching image, calculated as follows:

[0081]

[0082] Where n represents the number of images to be stitched, I(i) represents all images that match image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix. ij .

[0083] S9 obtains the relative positions of the images on the same plane based on the calculated transformation matrix of the images; this means transforming all target images into the coordinate system of the reference image. This step is obtained using the transformation matrix, thus completing the registration.

[0084] S10 constructs a spectral fitting function for the overlapping region based on the set of feature point pairs; the specific steps are as follows:

[0085] E(x,y)=a×m(x i ,y i )+b×m(x j ,y j )

[0086] Where m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j ) represents the original spectral value at pixel (x,y) in Figure j, and a and b are the spectral coefficients to be solved.

[0087] S11 obtains the corresponding spectral coefficients by minimizing the spectral error function. The specific calculation steps are as follows:

[0088]

[0089] Where M is the total number of matching point pairs, Δ k This represents the feature in-point pair (x) corresponding to the k-th matching pair. i ,y i ) and (x j ,y j The difference in spectral values, i.e.

[0090]

[0091] In the above formula and These represent the weighted spectral values ​​of the k-th feature pair, respectively, and their form is consistent with the spectral fitting function. By inputting the spectral values ​​of all feature pairs, the values ​​of the spectral coefficients a and b can be calculated.

[0092] S12 uses the obtained spectral coefficients to calculate the new spectral values ​​of pixels in the overlapping region, while the non-overlapping region remains unchanged, resulting in the final panoramic image. The specific calculation steps are as follows:

[0093]

[0094] In the above formula, m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j ) represents the original spectral value at pixel (x,y) in image j, and m′(x,y) represents the spectral value at pixel (x,y) in the final panoramic image.

[0095] Figure 2(a) shows the stitching effect using the traditional method, and Figure 2(b) shows the stitching effect using the proposed method. The traditional image stitching method for transformation and fusion results in images with problems such as ghosting and misalignment, as well as a certain degree of spectral distortion, resulting in poor stitching quality that cannot meet the needs of practical industrial applications. In contrast, the spectral fitting-based method proposed in this embodiment achieves better stitching results with less spectral distortion, indicating that this embodiment is more in line with practical application requirements than existing algorithms.

[0096] This embodiment also provides a storage medium storing a program that, when executed by a processor, implements a hyperspectral image stitching method based on spectral fitting.

[0097] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0098] This embodiment provides a hyperspectral image stitching method based on spectral fitting. SIFT feature points are extracted from each hyperspectral image, and SIFT feature points in multiple hyperspectral images are matched to obtain in-feature points. After transformation, a spectral fitting function for the overlapping region is constructed based on the set of in-feature point pairs. By minimizing the spectral fitting function, the corresponding spectral coefficients are obtained, and the new spectral values ​​of pixels in the overlapping region are calculated. Non-overlapping regions remain unchanged, and then the images are fused to obtain the final panoramic image, achieving a better stitching effect.

[0099] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A hyperspectral image stitching method based on spectral fitting, characterized in that, include: Read multiple hyperspectral images to be stitched together, and extract feature points from each hyperspectral image separately; Feature points in multiple hyperspectral images are matched to obtain feature matching point pairs. Based on the matching relationship between images and the feature intra-points obtained after filtering, a set of feature intra-points between multiple images is obtained. Calculate the transformation matrix between pairwise matched images; Select the image with the most matching images as the reference image, and the remaining images as the target images; The transformation matrix of the target image that does not directly overlap with the reference image is transformed into the transformation matrix in the coordinate system of the reference image by concatenation, while the transformation matrix of the target image that directly overlaps with the reference image remains unchanged. Iteratively optimize the transformation matrix parameters of the image and calculate the final transformation matrix; Based on the final transformation matrix of the image, the relative positions of the images on the same plane are obtained; Construct a spectral fitting function for the overlapping region based on the set of feature in-point pairs; The spectral coefficients are obtained by minimizing the spectral error function; The obtained spectral coefficients are used to calculate the new spectral values ​​of pixels in the overlapping region, while the non-overlapping region remains unchanged, resulting in the final panoramic image. Based on the set of feature in-point pairs, a spectral fitting function for the overlapping region is constructed as follows: E(x,y)=a×m(x i ,and i )+b×m(x j ,and j ) Where m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j ) represents the original spectral value at pixel (x,y) in Figure j, and a and b are the spectral coefficients to be solved; The spectral coefficients are obtained by minimizing the spectral error function, specifically as follows: Where M is the total number of matching point pairs, Δ k This represents the feature in-point pair (x) corresponding to the k-th matching pair. i ,y i ) and (x j ,y j The difference in spectral values, i.e. In the above formula and These represent the weighted spectral values ​​of the k-th feature pair, which are in the same form as the spectral fitting function. By inputting the spectral values ​​of all feature pairs, the values ​​of the spectral coefficients a and b can be obtained. The iterative optimization of the image transformation matrix parameters, and the calculation of the final transformation matrix, are as follows: The error between two matched images i and j is defined as the sum of the distances between the new coordinates of all feature points after transformation by the homography transformation matrix and the original coordinates. The calculation method is as follows: in, and Let k be the in-feature point of the feature in images i and j, respectively. H represents all feature interior points of images i and j. ij Let represent the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between each image and its matching image, calculated as follows: Where n represents the number of images to be stitched, I(i) represents all images that match image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix. ij .

2. The hyperspectral image stitching method according to claim 1, characterized in that, The obtained spectral coefficients are used to calculate the new spectral values ​​of pixels in the overlapping region, while the non-overlapping regions remain unchanged, resulting in the final panoramic image. In the above formula, m(x) i ,y i ) represents the original spectral value at pixel (x,y) in Figure i, m(x j ,y j ) represents the original spectral value at pixel (x,y) in image j, and m′(x,y) represents the spectral value at pixel (x,y) in the final panoramic image.

3. The hyperspectral image stitching method according to claim 1, characterized in that, The matching relationship between images, specifically, determines whether pairs of images match. The determination process is as follows: Let the total number of interior points of the feature be n. f The number of feature in-point pairs is n i If n i >8+0.3·n f If the two images match, then they are matched.

4. The hyperspectral image stitching method according to claim 1, characterized in that, The RANSAC algorithm was used to filter the feature points.

5. The hyperspectral image stitching method according to claim 1, characterized in that, Feature points in multiple hyperspectral images are matched to obtain feature in-place points between multiple images. Specifically, the ratio of nearest Euclidean distance to second nearest Euclidean distance is used to achieve coarse matching of feature descriptor vectors.

6. A system for implementing the hyperspectral image stitching method based on spectral fitting as described in any one of claims 1-5, characterized in that, include: Image acquisition module: Reads multiple hyperspectral images to be stitched together and extracts feature points from each hyperspectral image; Image matching module: Matches feature points in multiple hyperspectral images to obtain a set of feature points within the images; Image registration module: Calculates the transformation matrix between pairwise matching images, and transforms the transformation matrix of the target image that does not overlap with the reference image into the transformation matrix in the coordinate system of the reference image by concatenation, while the transformation matrix of the target image that directly overlaps with the reference image remains unchanged. Iteratively optimize the transformation matrix parameters of the image, calculate the final transformation matrix, and obtain the relative positions of the images on the same plane; Image fusion module: Based on the set of feature point pairs, construct a spectral fitting function for the overlapping region; obtain spectral coefficients by minimizing the spectral error function; use the obtained spectral coefficients to calculate the new spectral values ​​of pixels in the overlapping region, while keeping the non-overlapping region unchanged, to obtain the final panoramic image.

7. A storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the hyperspectral image stitching method based on spectral fitting as described in any one of claims 1-6.

Images