Hyperspectral image splicing method and system based on spectrum fitting

Abstract

The invention discloses a hyperspectral image splicing method and system based on spectrum fitting, and the method comprises the steps: respectively extracting feature points of each hyperspectral image, carrying out the matching of the feature points in a plurality of hyperspectral images, obtaining feature inner points, carrying out the transformation, constructing an overlapping region spectrum fitting function according to a feature inner point pair set, and carrying out the splicing of the hyperspectral images. And a corresponding spectrum coefficient is obtained by minimizing a spectrum fitting function, a new spectrum value of a pixel point in an overlapping region is calculated, a non-overlapping region is kept unchanged, a final panorama is obtained through fusion, and a better splicing effect is obtained.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image intelligent processing, and in particular to a hyperspectral image stitching method and system based on spectral fitting. Background Art

[0002] Hyperspectral remote sensing technology uses many narrow electromagnetic bands to capture detailed information of target objects, combines the spectral characteristics and spatial characteristics of the target, realizes the fusion of image and spectral information, and has both two-dimensional spatial information and one-dimensional spectral information. Many substances are difficult to detect in traditional wide-band remote sensing, but they are easier to detect if hyperspectral remote sensing is used. Therefore, hyperspectral remote sensing has great research value and development potential. Hyperspectral images have spectral characteristics, that is, different objects on the surface have different reflection and absorption characteristics for electromagnetic waves of specific wavelengths, and these characteristics are included in hyperspectral images; at the same time, hyperspectral images also have inter-spectral correlation, that is, at the same spatial position coordinates, different bands of hyperspectral images have certain similarities, because the reflectivity of the same object in different bands is often very close; and multiple identical pixels in different bands still represent the same object, so they have the same spatial topological structure. Image stitching technology is mainly divided into four steps, namely image acquisition, image preprocessing, image registration and image fusion. In the image fusion stage, due to the errors in the interpolation method and fusion strategy, the image may be blurred in the overlapping area, and the original image information cannot be accurately restored. In the process of aerial photography, although the time interval between adjacent images is not long, changes in weather, lighting, shadows, etc. will generally cause differences in color, brightness, etc. between images, and then cause spectral differences. After directly registering such images, obvious stitching gaps or abrupt transitions may occur in the overlapping area. Although some preprocessing methods can be used to make the grayscale between images tend to be consistent, this is a relatively rough and extensive adjustment. In the panorama, each pixel in the overlapping area is actually a matching point pair in at least two sub-images. In hyperspectral remote sensing images, each pixel has a fixed spectral value in each band. However, there are also certain differences in the spectral values ​​between the matching feature point pairs. In order to achieve seamless stitching of hyperspectral images, it is necessary not only to align the pixels in the overlapping area in spatial position, but also to minimize the spectral differences of the same pixel in different bands. Therefore, it is necessary to combine the image characteristics and develop new algorithms to deal with problems such as stitching seams and brightness differences.

[0003] Hyperspectral images have both spatial and spectral characteristics. A method that combines the above two characteristics, which can make good use of spatial information and add spectral information to assist in stitching, is proposed to solve the various problems faced. It has great practical and research value. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, an object of the present invention is to provide a hyperspectral image stitching method based on spectral fitting.

[0005] Another object of the present invention is to provide a hyperspectral image stitching system based on spectral fitting.

[0006] The present invention proposes a hyperspectral image stitching method based on spectral fitting. The method takes the set of matching point pairs in the overlapping area as a benchmark. In view of the problem that the spectral values ​​of the same object in the same band with the same wavelength may be different in reality, the spectral difference of the same pixel in different bands should be minimized as much as possible. The method performs image fusion based on the idea of ​​minimizing the spectral difference. The method adopts the strategy of minimizing the spectral difference of the matching point pairs in the overlapping area, constructs the spectral fitting function of the overlapping area, calculates and obtains the parameter value by minimizing the spectral error function, and then obtains the new spectral value of each pixel in the overlapping area, that is, the spectral value after fitting. The fusion step is completed, and the stitching effect is effectively improved.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A hyperspectral image stitching method based on spectral fitting, comprising:

[0009] Read multiple hyperspectral images to be stitched, and extract the feature points of each hyperspectral image respectively;

[0010] Match the feature points in multiple hyperspectral images to obtain feature matching point pairs, and obtain the feature inlier point set between multiple images based on the matching relationship between images and the feature inliers obtained after screening;

[0011] Calculate the transformation matrix between pairwise matching images;

[0012] Select the image with the largest number of matching images as the reference image, and the remaining images as the target images;

[0013] The transformation matrix of the target image that does not directly overlap with the reference image is transformed into the transformation matrix of the reference image coordinate system through cascading, and the transformation matrix of the target image that directly overlaps with the reference image remains unchanged;

[0014] Iteratively optimize the transformation matrix parameters of the image and calculate the final transformation matrix;

[0015] According to the final transformation matrix of the image, the relative position of the image on the same plane is obtained;

[0016] According to the characteristic inner point pair set, the spectral fitting function of the overlapping area is constructed;

[0017] By minimizing the spectral error function, the spectral coefficients are obtained;

[0018] The obtained spectral coefficients are used to calculate the new spectral values ​​of the pixels in the overlapping area, and the non-overlapping area remains unchanged to obtain the final panoramic image.

[0019] Furthermore, according to the characteristic interior point pair set, the overlapping area spectrum fitting function is constructed, specifically:

[0020] E(x,y)=a×m(x i ,y i )+b×m(x j ,y j )

[0021] Among them, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j ) represents the original spectral value at the pixel point (x, y) in Figure j, and a and b are the spectral coefficients to be solved.

[0022] Furthermore, by minimizing the spectral error function, the spectral coefficient is obtained, which is:

[0023]

[0024] Among them, M is the total number of matching point pairs, Δ k Indicates the feature inlier pair corresponding to the kth matching pair (x i ,y i ) and (x j ,y j ), that is,

[0025]

[0026] In the above formula and They represent the weighted spectral values ​​of the kth pair of feature interior point pairs respectively. Their form is consistent with the spectral fitting function. By inputting the spectral values ​​of all feature interior point pairs, the values ​​of the spectral coefficients a and b can be calculated.

[0027] Furthermore, the obtained spectral coefficients are used to calculate the new spectral values ​​of the pixels in the overlapping area, and the non-overlapping area remains unchanged to obtain the final panoramic image, which is specifically:

[0028]

[0029] In the above formula, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j) represents the original spectral value at the pixel point (x, y) in image j, and m′(x, y) represents the spectral value of the pixel point (x, y) in the final panoramic image.

[0030] Furthermore, the transformation matrix parameters of the image are iteratively optimized to calculate the final transformation matrix, specifically:

[0031] The error between two matching images i and j is defined as the sum of the distances between the new coordinate points of all feature internal points after being transformed by the homography transformation matrix and the original coordinate points. The calculation method is as follows:

[0032]

[0033] in, and are the kth feature inliers in image i and image j respectively, represents all the feature internal points of image i and image j, H ij represents the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between all images and their matching images, which is calculated as follows:

[0034]

[0035] Where n represents the number of images to be stitched, I(i) represents all images matching image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix ij .

[0036] Furthermore, the matching relationship between images is to determine whether the image pairs match each other. The judgment process is as follows:

[0037] Assume the total number of feature internal points is n f , the number of feature internal point pairs is n i , if n i >8+0.3·n f , then the two images match.

[0038] Furthermore, the RANSAC algorithm is used to screen the feature points.

[0039] Furthermore, feature points in multiple hyperspectral images are matched to obtain feature inliers between multiple images. Specifically, the ratio method of the nearest Euclidean distance and the second nearest Euclidean distance is used to achieve coarse matching of feature descriptor vectors.

[0040] A system for implementing the hyperspectral image stitching method based on spectral fitting, comprising:

[0041] Image acquisition module: read multiple hyperspectral images to be stitched and extract the feature points of each hyperspectral image;

[0042] Image matching module: matches the feature points in multiple hyperspectral images to obtain the feature inlier set between multiple images;

[0043] Image registration module: Calculate the transformation matrix between the two matching images, transform the transformation matrix of the target image that does not overlap with the reference image into the transformation matrix in the reference image coordinate system through cascading, and keep the transformation matrix of the target image that directly overlaps with the reference image unchanged; iteratively optimize the transformation matrix parameters of the image, calculate the final transformation matrix, and obtain the relative position of the image on the same plane;

[0044] Image fusion module: construct the spectral fitting function of the overlapping area according to the set of feature inliers; obtain the spectral coefficient by minimizing the spectral error function; use the obtained spectral coefficient to calculate the new spectral value of the pixel point in the overlapping area, and keep the non-overlapping area unchanged to obtain the final panoramic image.

[0045] A storage medium stores a program, and when the program is executed by a processor, the hyperspectral image stitching method based on spectral fitting is implemented.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] The method proposed in the present invention utilizes the spectral difference and consistency between feature internal points, adopts the strategy of minimizing the spectral difference of the matching feature internal point pairs in the overlapping area, constructs the spectral fitting function of the overlapping area, and calculates the value of the spectral coefficient by minimizing the function, thereby obtaining the new spectral value of each pixel in the overlapping area, and obtaining a better fusion effect. Compared with the traditional image stitching method, the stitching effect of the spectral fitting fusion method is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the hyperspectral image stitching method based on spectral fitting of the present invention;

[0049] FIG2(a) is a diagram showing the splicing effect using the traditional method, and FIG2(b) is a diagram showing the splicing effect using the present method;

[0050] Figure 3 It is a work flow chart of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0052] Example

[0053] like Figure 1-Figure 3 As shown, a hyperspectral image stitching method based on spectral fitting includes the following steps:

[0054] S1 reads multiple hyperspectral images to be stitched and extracts the feature points of each hyperspectral image;

[0055] S2 matches the feature points in multiple hyperspectral images to obtain a set of feature matching pairs between multiple images;

[0056] Specifically:

[0057] The ratio method of the nearest Euclidean distance and the second nearest Euclidean distance is used to achieve rough matching of feature descriptor vectors. Select a feature point in the reference image, then find the two feature points with the closest and second closest Euclidean distance to the feature point in the target image, and then divide the two Euclidean distance values. If the result is less than the set threshold, the pair of feature points with the closest Euclidean distance are considered to match.

[0058] Assume that the feature point descriptor R of the reference image and the feature point descriptor S of the target image are expressed as follows:

[0059] R=[r1,r2,…,r 128 ] T

[0060] S=[s1,s2,…,s 128 ] T

[0061] The Euclidean distance between the two is calculated as follows:

[0062]

[0063] After using the above formula to get the closest and second closest Euclidean distances, the ratio of the two can be used to determine whether the pair of feature points matches:

[0064]

[0065] Among them, threshold represents the threshold, and its size determines the number of feature matching pairs obtained.

[0066] The reference image in this step is local (valid only in the feature matching stage), which refers to the relationship between a pair of (two) images to be matched each time selected in the process of matching feature points. One of them is used as the reference image for feature point matching, and the other is used as the target image for feature point matching. The relationship between them is only valid between the pair of images in this step, and the relationship between the pair of images can be exchanged (the same image can be both a target image and a reference image), which does not affect the matching results.

[0067] S3 screens the obtained feature matching pairs to obtain feature inlier pairs between multiple image pairs, and uses the RANSAC algorithm to remove mismatched point pairs and retain the correct matching point pairs, which are the feature inlier pairs;

[0068] S4 determines whether the corresponding image pairs match based on the number of feature inliers. The specific steps are as follows: Assume that the total number of feature inliers is n f , the number of feature inliers obtained after screening by the RANSAC algorithm is n i , if n i >8+0.3·n f , then the two images match.

[0069] S5 calculates the transformation matrix between the two matching images according to the matching relationship of the matching images and the filtered feature inner point pairs. The specific steps are as follows: suppose one pair of inner points in the two images is [x1y1] T and [x2y2] T , transform both to the homogeneous coordinate system [x1y11] T and [x2y21] T After the coordinates are obtained, assume that the relationship between the two satisfies the homography transformation matrix H:

[0070]

[0071] By inputting the coordinates of all feature point pairs, the homography transformation matrix H can be obtained.

[0072] S6 selects an image with the largest number of matching images as a reference image, and the remaining images as target images.

[0073] The reference image in this step is more holistic and is mainly valid in the entire alignment process. It is selected from all the images to be stitched. Generally, there is only one reference image. The reference image is the base coordinate system for the transformation, so it does not need to be transformed; the remaining images to be stitched that are not reference images are all target images that need to be transformed.

[0074] S7 transforms the transformation matrix of the target image that does not directly overlap with the reference image into the transformation matrix of the reference image coordinate system through cascading, and the transformation matrix of the target image that directly overlaps with the reference image remains unchanged;

[0075] Further explanation: Generally speaking, if there are more than four matching pairs of feature inliers between two images, the two images are considered to have an overlapping area. Otherwise, the images are considered to have no overlap or the overlapping area is too small to be calculated. The feature inliers are all in the overlapping area of ​​the images.

[0076] The feature extraction algorithm is for a single image, and what is obtained is feature points evenly distributed on each image (including overlapping areas and non-overlapping areas); while the feature matching algorithm is for the relationship between two images. It uses the feature points of the two images as input to determine whether there is an identical matching relationship (that is, the transformation matrix) between these feature points. If there is, it is a feature inlier, and if not, it is an exlier. The inliers are all located in the overlapping area, because generally only the feature points in the overlapping area can be matched.

[0077] The cascade means that the transformation relationship between two images needs to be calculated through the transformation moments of other images that overlap with the two images respectively.

[0078] S8 iteratively optimizes the transformation matrix parameters of the image and calculates the final transformation matrix. The specific steps are: the error between two matching images i and j is defined as the sum of the distances between the new coordinate points of all feature internal points after being transformed by the homography transformation matrix and the original coordinate points. The calculation method is as follows:

[0079]

[0080] in, and are the kth feature inliers in image i and image j respectively, represents all the feature internal points of image i and image j, H ij represents the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between all images and their matching images, calculated as follows:

[0081]

[0082] Where n represents the number of images to be stitched, I(i) represents all images matching image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix ij .

[0083] S9 obtains the relative position of the image on the same plane according to the calculated transformation matrix of the image; it means that all target images are transformed into the coordinate system of the reference image. This step is obtained by using the transformation matrix, and the registration is completed.

[0084] S10 constructs a spectral fitting function for the overlapping region according to the characteristic interior point pair set; the specific steps are:

[0085] E(x,y)=a×m(x i ,y i )+b×m(x j ,y j )

[0086] Among them, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j ) represents the original spectral value at the pixel point (x, y) in Figure j, and a and b are the spectral coefficients to be solved.

[0087] S11 obtains the corresponding spectral coefficient by minimizing the spectral error function. The specific calculation steps are:

[0088]

[0089] Among them, M is the total number of matching point pairs, Δ k Indicates the feature inlier pair corresponding to the kth matching pair (x i ,y i ) and (x j ,y j ), that is,

[0090]

[0091] In the above formula and They represent the weighted spectral values ​​of the kth pair of feature inliers, and their form is consistent with the spectral fitting function. By inputting the spectral values ​​of all feature inliers, the values ​​of spectral coefficients a and b can be obtained.

[0092] S12 uses the obtained spectral coefficients to calculate the new spectral values ​​of the pixels in the overlapping area, and the non-overlapping area remains unchanged to obtain the final panoramic image. The specific calculation steps are:

[0093]

[0094] In the above formula, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j ) represents the original spectral value at the pixel point (x, y) in image j, and m′(x, y) represents the spectral value of the pixel point (x, y) in the final panoramic image.

[0095] FIG2(a) is a stitching effect diagram using the traditional method, and FIG2(b) is a stitching effect diagram using the present method. When the traditional image stitching method is used for transformation and fusion, the resulting image has problems such as ghosting and misalignment, and the spectrum also has a certain degree of distortion. The stitching quality is poor and cannot meet the actual industrial application requirements. The method based on spectral fitting proposed in this embodiment has a good stitching effect and the spectral distortion of the image is small, which shows that this embodiment is more in line with the actual application requirements than the existing algorithm.

[0096] This embodiment also provides a storage medium storing a program, which, when executed by a processor, implements a hyperspectral image stitching method based on spectral fitting.

[0097] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0098] This embodiment provides a hyperspectral image stitching method based on spectral fitting, which extracts SIFT feature points of each hyperspectral image respectively, matches SIFT feature points in multiple hyperspectral images, obtains feature inliers and then transforms them, constructs a spectral fitting function for the overlapping area based on a set of feature inliers, and obtains corresponding spectral coefficients by minimizing the spectral fitting function, calculates new spectral values ​​of pixel points in the overlapping area, and keeps non-overlapping areas unchanged, and then fuses them to obtain the final panoramic image, thereby obtaining a better stitching effect.

[0099] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A hyperspectral image stitching method based on spectral fitting, characterized in that: include: Read multiple hyperspectral images to be stitched, and extract the feature points of each hyperspectral image respectively; Match the feature points in multiple hyperspectral images to obtain feature matching point pairs, and obtain the feature inlier point set between multiple images based on the matching relationship between images and the feature inliers obtained after screening; Calculate the transformation matrix between pairwise matching images; Select the image with the largest number of matching images as the reference image, and the remaining images as the target images; The transformation matrix of the target image that does not directly overlap with the reference image is transformed into the transformation matrix of the reference image coordinate system through cascading, and the transformation matrix of the target image that directly overlaps with the reference image remains unchanged; Iteratively optimize the transformation matrix parameters of the image and calculate the final transformation matrix; According to the final transformation matrix of the image, the relative position of the image on the same plane is obtained; According to the characteristic inner point pair set, the spectral fitting function of the overlapping area is constructed; By minimizing the spectral error function, the spectral coefficients are obtained; The obtained spectral coefficients are used to calculate the new spectral values ​​of the pixels in the overlapping area, and the non-overlapping area remains unchanged to obtain the final panoramic image.

2. The hyperspectral image stitching method according to claim 1, characterized in that: According to the characteristic inner point pair set, the overlapping area spectrum fitting function is constructed, which is as follows: E(x,y)=a×m(x i ,and i )+b×m(x j ,and j ) Among them, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j ) represents the original spectral value at the pixel point (x, y) in Figure j, and a and b are the spectral coefficients to be solved.

3. The hyperspectral image stitching method according to claim 2, characterized in that: By minimizing the spectral error function, the spectral coefficient is obtained, which is: Among them, M is the total number of matching point pairs, Δ k Indicates the feature inlier pair corresponding to the kth matching pair (x i ,y i ) and (x j ,y j ), that is, In the above formula and They represent the weighted spectral values ​​of the kth pair of feature interior point pairs respectively. Their form is consistent with the spectral fitting function. By inputting the spectral values ​​of all feature interior point pairs, the values ​​of the spectral coefficients a and b can be calculated.

4. The hyperspectral image stitching method according to claim 3, characterized in that: The obtained spectral coefficients are used to calculate the new spectral values ​​of the pixels in the overlapping area, and the non-overlapping area remains unchanged to obtain the final panoramic image, which is as follows: In the above formula, m(x i ,y i ) represents the original spectral value at the pixel point (x, y) in Figure i, m(x j ,y j ) represents the original spectral value at the pixel point (x, y) in image j, and m′(x, y) represents the spectral value of the pixel point (x, y) in the final panoramic image.

5. The hyperspectral image stitching method according to claim 1, characterized in that: The iterative optimization of the transformation matrix parameters of the image and calculation of the final transformation matrix are specifically as follows: The error between two matching images i and j is defined as the sum of the distances between the new coordinate points of all feature internal points after being transformed by the homography transformation matrix and the original coordinate points. The calculation method is as follows: in, and are the kth feature inliers in image i and image j respectively, represents all the feature internal points of image i and image j, H ij represents the homography transformation matrix between image i and image j. The cumulative error of all images is the sum of the errors between all images and their matching images, which is calculated as follows: Where n represents the number of images to be stitched, I(i) represents all images matching image i, and then iterative optimization is performed to calculate the parameters H of the homography transformation matrix ij .

6. The hyperspectral image stitching method according to claim 1, characterized in that: The matching relationship between images is to determine whether the image pairs match each other. The judgment process is as follows: Assume the total number of feature internal points is n f , the number of feature internal point pairs is n i , if n i >8+0.3·n f , then the two images match.

7. The hyperspectral image stitching method according to claim 1, characterized in that: The RANSAC algorithm is used to screen the feature points.

8. The hyperspectral image stitching method according to claim 1, characterized in that: The feature points in multiple hyperspectral images are matched to obtain the feature inliers between multiple images. Specifically, the ratio method of the nearest Euclidean distance and the second nearest Euclidean distance is used to achieve the coarse matching of the feature descriptor vectors.

9. A system for implementing the hyperspectral image stitching method based on spectral fitting according to any one of claims 1 to 8, characterized in that: include: Image acquisition module: read multiple hyperspectral images to be stitched and extract the feature points of each hyperspectral image; Image matching module: matches the feature points in multiple hyperspectral images to obtain the feature inlier set between multiple images; Image registration module: Calculate the transformation matrix between the two matching images, transform the transformation matrix of the target image that does not overlap with the reference image into the transformation matrix of the reference image coordinate system through cascading, and the transformation matrix of the target image that directly overlaps with the reference image remains unchanged; Iteratively optimize the transformation matrix parameters of the image, calculate the final transformation matrix, and obtain the relative position of the image on the same plane; Image fusion module: construct the spectral fitting function of the overlapping area according to the set of feature inliers; obtain the spectral coefficient by minimizing the spectral error function; use the obtained spectral coefficient to calculate the new spectral value of the pixel point in the overlapping area, and keep the non-overlapping area unchanged to obtain the final panoramic image.

10. A storage medium storing a program, characterized in that: When the program is executed by a processor, the hyperspectral image stitching method based on spectral fitting according to any one of claims 1 to 8 is implemented.

Images