Multi-feature fusion and adaptive optimization SIFT image splicing method and system

Abstract

The application discloses a SIFT image splicing method and system based on multi-feature fusion and adaptive optimization, and comprises the following steps: performing data preprocessing on acquired image data to generate a gray image; constructing a scale space by using a SIFT feature extraction method, detecting local extreme points in the gray image to generate a descriptor, and performing SIFT feature matching optimization; performing false matching elimination by using a nearest neighbor ratio criterion and a matching threshold value, and maintaining the distribution balance of feature points in the gray image by using a uniform sampling method; performing random sample consensus estimation by using an adaptive threshold value in a feature matching point set, eliminating outliers, and calculating a homography matrix to realize image registration; and realizing image splicing in an overlapping area based on a weighted fusion strategy, avoiding the abruptness of a joint transition by combining a smoothing weight, and performing black edge trimming to optimize a splicing result. The application can optimize a traditional SIFT image splicing algorithm, and thus the splicing quality of an image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image stitching, in particular to a multi-feature fusion and adaptive optimization SIFT image stitching method and system. BACKGROUND

[0002] Image stitching technology generates a complete panoramic image by fusing multiple overlapping images, and is widely used in panoramic photography, medical image stitching (CT / MR, cultural relic digital reconstruction, remote sensing image processing, etc.

[0003] At present, the SIFT (Scale Invariant Feature Transform) algorithm, as a widely used classical method in image stitching, has become the mainstream feature extraction and matching scheme in the prior art because of its scale invariance and rotation invariance, which can adapt to the change of light to a certain extent and effectively extract image features.

[0004] However, in actual application, when the texture in the scene is sparse (such as the sky or a single-color wall) or there is an occlusion, relying only on a single SIFT feature can easily lead to insufficient number of feature points and increased false matching rate, thereby causing deviation in subsequent geometric transformation estimation; in terms of feature point distribution, the traditional SIFT matching result often has feature point redundancy and local concentration, and even repeated matching points, which increases the calculation amount of the RANSAC algorithm and reduces the overall registration efficiency; in terms of parameter setting, the existing SIFT scheme usually uses a fixed matching threshold and RANSAC inlier threshold, lacks an adaptive adjustment mechanism, and is difficult to optimize for different image texture densities and overlap degrees, resulting in insufficient stitching accuracy in some scenes; in addition, the prior art lacks degradation compatibility, for example, when SURF and other feature algorithms are unavailable due to the running environment, there is a lack of mechanism to automatically switch to a combination scheme of SIFT and ORB, thereby affecting the stability and practicality of the system; at the same time, in the fusion and post-processing link of the stitching result, some methods fail to perform histogram matching on the brightness difference of the overlapping area, and also fail to use feathering fusion and other means to eliminate the joint, and the black edge cropping strategy is relatively rough, resulting in poor overall visual quality of the final stitched image.

[0005] Therefore, there is an urgent need for a SIFT image stitching method that has multi-feature stability, adaptive optimization, degradation compatibility, and high-quality fusion. SUMMARY

[0006] The purpose of the present application is to overcome the above shortcomings, and the purpose of the present application is to provide a multi-feature fusion and adaptive optimization SIFT image stitching method and system, to solve the problem of insufficient robustness of single SIFT feature matching, to improve the adaptation ability in complex scenes through multi-feature fusion, to realize feature point de-redundancy and uniform sampling, to reduce the calculation amount and to improve the registration accuracy.

[0007] To solve the above technical problems, the application provides a multi-feature fusion and adaptive optimization SIFT image stitching method, comprising:

[0008] S1: data preprocessing is performed on the acquired image data to generate a gray image;

[0009] S2: a scale space is constructed by using a SIFT feature extraction method, local extreme points are detected in the gray image, a descriptor is generated, and then SIFT feature matching optimization is performed;

[0010] S3: false matching is removed by using a nearest neighbor ratio criterion and a matching threshold, and the distribution uniformity of feature points in the gray image is maintained by using a uniform sampling method, so as to reduce the calculation amount;

[0011] S4: in the feature matching point set, a random sample consensus estimation is performed by using an adaptive threshold, outliers are removed, and a homography matrix is calculated, so as to realize image registration;

[0012] S5: image stitching is realized in the overlapping area based on a weighted fusion strategy, a smooth weight is combined to avoid the abrupt transition of a joint, and after the stitching is completed, black edge cropping is performed to optimize the stitching result.

[0013] As a preferred mode of the application, in step S1, the method comprises:

[0014] S11: at least two RGB input images are read according to the collected image data and are defined as , wherein , is the number of image pixel rows, is the number of image pixel columns, is the data type of the image pixel value, which is an integer;

[0015] S12: gray processing is performed:

[0016] , is the pixel value of the i-th row, the j-th column and the first channel, is the pixel value of the i-th row, the j-th column and the second channel, is the pixel value of the i-th row, the j-th column and the third channel, is a down rounding operation;

[0017] S13: the generated gray image is .

[0018] As a preferred mode of the application, in step S2, the method comprises:

[0019] S21: Construct the scale space of the image by the method of Gaussian pyramid, ensure that the feature points can be detected at different scales, wherein the scale space is defined as , the calculation formula is: , is the input gray image, is two-dimensional convolution, is the Gaussian kernel function, , is the scale parameter;

[0020] S22: Calculate the Gaussian difference image between adjacent scales: , wherein is the scale factor of adjacent scales, is the scale space function; then compare each pixel in with the surrounding 8 pixels of the same scale and 18 pixels of the adjacent scales above and below, if it is a local extreme value, it is considered as a candidate feature point;

[0021] S23: For the detected candidate feature point, extract the neighborhood region of 16x16 around the feature point, and then calculate the gradient amplitude and direction of each pixel point in the neighborhood:

[0022] ,

[0023] , is the gradient amplitude of the pixel point, is the pixel value of the scale space image L at the coordinate (x+1, y), is the pixel value of the scale space image L at the coordinate (x-1, y), is the pixel value of the scale space image L at the coordinate (x, y+1), is the pixel value of the scale space image L at the coordinate (x, y-1), is the gradient direction of the pixel point;

[0024] S24: Construct the gradient direction histogram, count the gradient direction in the neighborhood, and select the peak value direction in the histogram as the main direction of the feature point; in order to realize the scale invariance, divide the 16x16 neighborhood into 4x4 sub-blocks, count the gradient histogram of 8 directions in each sub-block, and finally generate a 4x4x8=128-dimensional descriptor.

[0025] As a preferred mode of the present application, in step S3, the method comprises:

[0026] S31: Use the Euclidean distance as the distance measurement standard of the descriptor, the calculation formula is:

[0027] ,in, To describe the sub-dimension, For two descriptors, Let Euclidean distance be the distance between the two descriptors. It is the dimension index of the descriptor;

[0028] S32: Feature matching using the nearest neighbor ratio criterion: for descriptors Find the two closest descriptors in the first input image. and ,like Then, retain and The match, where for Indicates the first input image and the descriptor The corresponding feature point coordinates, yes If the corresponding feature point is selected, it is considered a mismatch and discarded.

[0029] Among them, setting matching constraints ,in, For descriptor With the first input image descriptor The Euclidean distance between them Only points with a distance of less than 50 are allowed to be retained;

[0030] S33: The grayscale image is divided into grids of a preset size using uniform sampling. The coordinate range formula for the grid is:

[0031] For each grid cell, coordinates that satisfy... and The points are saved to provide data for subsequent registration; among them, Let be the coordinates of the left boundary column of the i-th grid. Let be the coordinates of the right boundary column of the i-th grid. Let j be the coordinates of the upper boundary row of the raster. Let j be the coordinates of the lower boundary row of the raster. The number of grid divisions.

[0032] In a preferred embodiment of this application, step S4 includes:

[0033] S41: Let the points in the second input image be... homogeneous coordinates Let the corresponding points in the first input image be... homogeneous coordinates ,but: wherein is a 3x3 homography matrix, , expanding its second equation can get , ;

[0034] S42: find the optimal homography matrix by using RANSAC registration method .

[0035] As a preferred mode of the present application, in step S42, the method comprises:

[0036] S421: define the inlier point threshold wherein , is the Euclidean distance corresponding to all matched points, , is the standard deviation of the distance;

[0037] S422: randomly select 4 points from the matched points, substitute the 4 points into the projection transformation formula to generate 8 linear equations, and calculate the inlier point number of the nth point , the calculation process is as follows:

[0038] ;

[0039] S423: solve by singular value decomposition to get .

[0040] As a preferred mode of the present application, in step S42, the method further comprises: for all matched points, calculating the predicted point after transformation , if , it is regarded as an inlier point, and the inlier point number is counted , if the current inlier point number is greater than the historical maximum value, the optimal is updated as the current model. As a preferred mode of the present application, in step S5, the method comprises:

[0041] S51: generate a smooth weight for the overlapping area between images

[0042] : , use the Gaussian kernel smooth weight with standard deviation to avoid the transition being too hard; and then expand to 3 channels, and the fusion formula is as follows:

[0043] wherein , is the coordinate variable of the image pixel, ​the result of the dilation operation on the overlap region mask M, = 1, the pixel of the second input image is completely reserved, = 0, the pixel of the first input image is completely reserved, 0 < 1, linear weighting, realizing smooth transition;

[0044] S52: the first input image and the second input image are realized through the smooth weight matrix transition, the overlap region is a region where both the first input image and the second input image have pixels, and the mask overlapMask is calculated according to the formula:

[0045] , wherein is the first input image, is the second input image, indicates that the first input image has valid pixels at .

[0046] As a preferred mode of the present application, after the image stitching is completed, the method further comprises:

[0047] S501: the stitching quality is quantified by using the root mean square error:

[0048] , wherein is the number of inliers; is the inlier coordinate of the first input image; is the inlier coordinate of the second input image after the transformation;

[0049] S502: the stitching quality is quantified by using the structural similarity index:

[0050] , wherein is the window pixel of the stitched image, is the corresponding window pixel of the first input image; is the mean of the window pixel ; is the mean of the window pixel ; is the standard deviation of the window pixel ; is the standard deviation of the window pixel ; is the covariance of the window pixel and ; , , is a constant to avoid the denominator being 0; is the total number of window pixels;​

[0051] S503: Quantify the splicing quality by using the peak signal-to-noise ratio: , wherein is the image bit depth; MSE is the mean square error, and only the pixel difference of the overlapping area is calculated, and the formula is , is the splicing overlapping area, is the first input image overlapping area;

[0052] S504: Quantify the splicing quality by using the edge transition smoothness: , wherein , is the gray image of , is the edge pixel set of the overlapping area, calculate the standard deviation of the gray difference of the edge pixels.

[0053] The application also provides a multi-feature fusion and adaptive optimization SIFT image splicing system using the above method, comprising:

[0054] a data processing module for acquiring image data and pre-processing the acquired image data to generate a gray image;

[0055] a feature matching module for constructing a scale space and detecting local extreme points in the gray image using the SIFT feature extraction method, generating a descriptor, and then performing SIFT feature matching optimization; performing false match elimination by using the nearest neighbor ratio criterion and the matching threshold, and using a uniform sampling method to maintain the distribution balance of feature points in the gray image, thereby reducing the computational complexity;

[0056] an image registration module for using adaptive thresholding to perform random sample consensus estimation in the feature matching point set, eliminating outliers and calculating the homography matrix to achieve image registration;

[0057] a splicing optimization module for implementing image splicing based on a weighted fusion strategy in the overlapping area and combining a smoothing weight to avoid abrupt transitions at the seam, and performing black border cropping after splicing to optimize the splicing result.

[0058] The above technical solutions of the application have the following advantages compared with the prior art:

[0059] 1. The application not only uses the scale invariance and rotation invariance of the SIFT descriptor, but also introduces false match elimination and uniform sampling mechanism in the feature matching process to avoid the problem of excessive concentration or redundancy of feature points, thereby improving the adaptation ability in complex scenes such as illumination change, scale transformation and local occlusion.

[0060] 2、By the near neighbor criterion combined with matching threshold to eliminate false matches, and combined with the uniform sampling method based on grid division, the application can reduce the number of redundant feature points under the premise of ensuring the coverage of feature points, thereby significantly reducing the computational complexity and improving the efficiency of image stitching processing.

[0061] 3、The application introduces an adaptive threshold setting method, which determines the threshold by dynamically calculating the standard deviation of the Euclidean distance of matching points, improves the accuracy of inlier selection; at the same time, singular value decomposition is used to solve the homography matrix and the optimal model is updated by iteration, which improves the accuracy of image registration.

[0062] 4、The application uses a weighted fusion strategy based on Gaussian smoothing to realize image transition through a smooth weight matrix, avoiding the edge abruptness problem caused by simple superposition, so that the transition of the stitching area is more natural and soft, and the overall visual effect is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0064] Figure 1 is the first flowchart of the SIFT image stitching method provided by the embodiment of the application.

[0065] Figure 2 is the second flowchart of the SIFT image stitching method provided by the embodiment of the application.

[0066] Figure 3 is the flowchart of the stitching quality evaluation method provided by the embodiment of the application.

[0067] Figure 4 is the feature point matching diagram of the method of the application and the prior art; wherein (a) is the SIFT feature point matching diagram, (b) is the ORB feature point matching diagram, (c) is the SURF feature point matching diagram, and (d) is the feature point matching diagram of the method of the application.

[0068] Figure 5 is the matching result diagram of the method of the application and the prior art; wherein (a) is the SIFT matching result diagram, (b) is the ORB matching result diagram, (c) is the SURF matching result diagram, and (d) is the matching result diagram of the method of the application.

[0069] Figure 6is a schematic diagram of an evaluation index of the method provided by the embodiment of the application and the prior art method.

[0070] Figure 7 is a schematic diagram of module connection of the SIFT image stitching system provided by the embodiment of the application. DETAILED DESCRIPTION

[0071] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0072] IFT (Scale-Invariant Feature Transform) algorithm is a classical local feature extraction method. The algorithm can stably detect and describe image feature points under the conditions of scale change, rotation change and a certain degree of illumination and view angle change, and is therefore widely used in image stitching, target recognition, three-dimensional reconstruction and image retrieval fields.

[0073] The core idea of the SIFT algorithm is to find stable feature points in the scale space and generate descriptors with scale invariance and rotation invariance for these feature points, so as to ensure that the image can still achieve robust feature matching under changes such as scaling, rotation and illumination. The main processing flow includes scale space construction, feature point detection, feature point accurate positioning, direction assignment, feature descriptor generation and feature matching.

[0074] In the scale space construction, in order to detect significant feature points under different scales, SIFT first constructs a Gaussian pyramid for the image. By performing multi-layer Gaussian blur and down-sampling on the input image, different scale image representations are obtained, so as to realize scale invariance in a mathematical sense.

[0075] In the feature point detection, in the scale space, potential feature points are detected by calculating the difference (i.e. Difference of Gaussian, DOG) of adjacent scale images; SIFT finds local extreme points (i.e. maximum or minimum in space and scale) in the DOG space as candidate feature points, so as to ensure the stability of the feature points under scale change.

[0076] In the feature point accurate positioning, interpolation and edge response suppression are performed on the candidate feature points to remove feature points with low contrast or located at edges, so as to improve the stability of the feature points.

[0077] In orientation assignment, to achieve rotation invariance, SIFT calculates the distribution of pixel gradient directions in the neighborhood of a feature point and assigns one or more dominant orientations to each feature point; this step ensures that the description of a feature point remains consistent when the image is rotated.

[0078] In feature descriptor generation, the gradient direction and magnitude information of the neighborhood (usually a 16x16 window) of a feature point is extracted and divided into several sub-regions, and the direction histogram is counted in each sub-region; finally, these histograms are combined into a 128-dimensional feature vector, i.e., the SIFT descriptor.

[0079] In feature matching, SIFT usually judges the similarity of feature points by calculating the Euclidean distance between feature descriptors, and often introduces a nearest neighbor ratio criterion (i.e., requiring the distance ratio of the nearest neighbor to the second nearest neighbor to be less than a certain threshold) to eliminate most false matches.

[0080] Thus, with reference to Figure 1 and Figure 2 , in some embodiments of the present application, a multi-feature fusion and adaptive optimization SIFT image stitching method is involved, which comprises the following steps:

[0081] S1: Preprocess the acquired image data to generate a grayscale image.

[0082] Specifically, in step S1, the method comprises the following steps:

[0083] S11: Read at least two RGB color images to be stitched from the image data of the acquisition device (such as a camera) or the image database and define them as , where , i.e. is a three-dimensional integer matrix , is the number of image pixel rows (image height), is the number of image pixel columns (image width), is the data type of the image pixel value, which is an integer; in the three-dimensional integer matrix, represents the pixel value of the i-th row, j-th column, and k-th channel, where k=1, 2, 3 corresponds to the red channel, green channel, and blue channel, respectively.

[0084] S12: To simplify the subsequent feature point detection and descriptor calculation, convert the read image to a grayscale image:

[0085] , is the pixel value of the i-th row, j-th column, and 1st channel, is the pixel value of the i-th row, j-th column, and 2nd channel, is the pixel value of the i-th row, j-th column, 3rd channel, is the pixel value of the i-th row, j-th column, 3rd channel,

[0086] S13: The generated gray image is .

[0087] After the above gray processing, the obtained gray image is denoted as , which belongs to a two-dimensional integer matrix , wherein represents the gray value of the i-th row, j-th column pixel, and the gray image will be used as the input data for subsequent multi-feature extraction and fusion matching.

[0088] Through the above gray processing, the original RGB image can be uniformly converted into a gray image, which not only reduces the data dimension and computational complexity, but also provides a unified and stable input basis for subsequent feature point detection and descriptor generation, thereby improving the overall efficiency of the image stitching method.

[0089] S2: A scale space is constructed using the SIFT feature extraction method, and local extreme points are detected in the gray image to generate a descriptor, and then SIFT feature matching optimization is performed.

[0090] Specifically, in step S2, the method comprises the following steps:

[0091] S21: In order to ensure that the feature points can be effectively detected at different scales, a Gaussian pyramid method is used to construct the scale space of the image to ensure that the feature points can be detected at different scales; specifically, after the input image is converted into a gray image, the gray image is used as the basis to perform Gaussian filtering at different scales. The definition of the scale space is , and the calculation formula is: , wherein is the input gray image, and the symbol represents two-dimensional convolution; is a Gaussian kernel function, and the Gaussian kernel function is defined as:

[0092] , is a scale parameter, and by adjusting the parameter, images with different degrees of blur can be generated, thereby obtaining a Gaussian pyramid composed of multi-level blurred images.

[0093] S22: After the scale space is constructed, a Difference of Gaussian (DoG) image between adjacent scales is calculated, and the calculation formula is as follows: wherein is a scale factor of adjacent scales, is a scale space function;

[0094] Then, in the difference image , each pixel point and the pixels in the surrounding neighborhood thereof are compared, specifically, 8 neighborhood pixels of the same scale and 18 neighborhood pixels of each of the two adjacent scales above and below are compared. If the pixel is a local maximum value or a local minimum value in the comparison result, the point is taken as a candidate feature point. In this way, non-stable feature points can be effectively excluded.

[0095] S23: For the detected candidate feature point, a 16x16 neighborhood region around the feature point is extracted with the feature point as the center; then, for each pixel point in the neighborhood, the gradient amplitude and direction thereof are calculated, and the specific calculation method is as follows:

[0096] Gradient amplitude:

[0097] ,

[0098] Gradient direction: ; is the gradient amplitude of the pixel point, is the pixel value of the scale space image L at the coordinate (x+1, y), is the pixel value of the scale space image L at the coordinate (x-1, y), is the pixel value of the scale space image L at the coordinate (x, y+1), is the pixel value of the scale space image L at the coordinate (x, y-1), is the gradient direction of the pixel point;

[0099] The local change characteristics in the neighborhood of the feature point are described through this step, which serves as the basis for subsequent direction assignment and descriptor generation.

[0100] S24: After the gradient information of the pixels in the neighborhood is obtained, a gradient direction histogram is constructed, the gradient directions in the neighborhood are counted, and the peak value direction in the histogram is selected as the main direction of the feature point; in this way, the rotation invariance of the feature is realized.

[0101] Further, in order to generate a feature descriptor with scale and rotation invariance, a 16x16 neighborhood is divided into 4x4 sub-regions; for each sub-region, a gradient histogram of 8 directions is counted respectively, thereby obtaining a direction feature vector with a length of 8; then the direction feature vectors of the 4x4 sub-regions are spliced to generate a 128-dimensional SIFT feature descriptor, which has strong distinguishability and can be effectively used for subsequent feature point matching and optimization.

[0102] S3: false matching elimination by means of a nearest neighbor ratio criterion and a matching threshold and uniform sampling to maintain the distribution balance of feature points in the gray-scale image, thereby reducing the calculation amount.

[0103] Specifically, in step S3, the method comprises the following steps:

[0104] S31: in order to measure the similarity between feature descriptors, the Euclidean distance is used as the distance measurement standard of the descriptor, and the specific formula is:

[0105] wherein, is the dimension of the descriptor (in this embodiment, the dimension of the SIFT descriptor is 128), are two descriptors to be compared respectively, is the Euclidean distance between the two descriptors, is the dimension index of the descriptor; by means of the above-mentioned Euclidean distance formula, the difference degree of two feature points in the feature space can be effectively measured.

[0106] S32: for each descriptor to be matched , the two closest descriptors in the feature point set of the other image are found, and are denoted as and , wherein is the nearest neighbor point, is the second nearest neighbor point, and then is calculated, if , i.e. if the ratio is less than a preset maximum ratio threshold , it is determined that the matching of and is valid, and the corresponding feature point matching relationship (i.e. the matching of the point pair and ) is retained; if the condition is not satisfied, it is considered as false matching and is eliminated.

[0107] In step S32, a distance constraint condition is also set: , wherein is the distance between the descriptor and the first input image descriptor Euclidean distance between them, , i.e. only when the Euclidean distance between descriptors is less than 50, the matching pair is allowed to be reserved. In this way, abnormal or unreliable matching point pairs can be further excluded.

[0108] S33: After the elimination of false matches, if all the remaining feature points are directly used for registration, there may be a problem that feature points are dense in some areas and sparse in other areas, thereby affecting the balance of registration and computational efficiency. Therefore, in this step, a uniform sampling method is used to maintain the balanced distribution of feature points in the entire gray image. Specifically, the input gray image is divided into several regular grids according to a predetermined grid number The coordinate range of each grid is defined as:

[0109] , where W and H are the width and height of the image respectively, and i, j are the row and column indices of the grid; in each grid range, all feature points satisfying and are selected and saved according to the set sampling strategy (such as random selection or preferential selection of feature points with higher matching quality), thereby ensuring that the finally retained feature points are uniformly distributed in the entire image range; wherein is the left boundary column coordinate of the i-th column grid, is the right boundary column coordinate of the i-th column grid, is the upper boundary row coordinate of the j-th row grid, is the lower boundary row coordinate of the j-th row grid, is the number of grid division.

[0110] S4: The random sample consensus (RANSAC, Random Sample Consensus) method with adaptive threshold is used to optimize the feature matching points, eliminate outliers and obtain the optimal homography matrix, thereby realizing accurate registration of two images.

[0111] In step S4, the method comprises:

[0112] S41: Assuming that the feature points in the second input image are , the homogeneous coordinate representation is , and the corresponding points in the first input image are , the homogeneous coordinate representation is , then the following relationship is satisfied between them: , wherein is a 3x3 homography matrix, defined as: , after expansion, we can get , ; Through the above formula, the points in the second input image can be projected into the coordinate system of the first input image.

[0113] S42: In this step, the RANSAC method is used to estimate the optimal homography matrix in the presence of outliers , specifically including the following sub-steps:

[0114] Specifically, in step S42, the method includes:

[0115] S421: Define the inlier threshold as , which is defined as: , where is the set of Euclidean distances corresponding to all matched points, defined as: , is the standard deviation of the distances of all matched points; through adaptive threshold setting, the discrimination condition can be dynamically adjusted according to the data distribution.

[0116] S422: Randomly select 4 pairs of points from all matched points and substitute them into the projection transformation formula to establish a constraint equation for the homography matrix ; for the th matched point , the calculation process is formula:

[0117] ; 4 pairs of points can generate 8 independent linear equations, thereby constructing a homogeneous equation system.

[0118] S423: Use the singular value decomposition (SVD) method to solve the above equation system to obtain the candidate homography matrix .

[0119] Specifically: based on the candidate matrix , project the points of the second input image into the coordinate system of the first input image to obtain the predicted points , which is defined as: , then calculate the error between the predicted points and the actual points: , if , then the point is considered as an inlier. Count the number of all inliers , if the number of inliers of the current candidate model is greater than the historical maximum value, then update the optimal homography matrix as the current solution.

[0120] Through multiple iterations of the above process, the homography matrix model containing the most inliers can be finally obtained, thereby realizing accurate registration between images.

[0121] S5: image stitching is implemented in the overlapping region based on a weighted fusion strategy, and a smooth weight is combined to avoid the seam transition from being conspicuous, and after the stitching is completed, black border cropping is performed to optimize the stitching result.

[0122] In step S5, the method comprises:

[0123] S51: in the input two registered images, a smooth weight matrix is first generated for the overlapping region thereof , and the calculation formula is: , wherein is a standard deviation of a Gaussian kernel smooth weight; is the result of an inflation operation on the overlapping region mask , and the weight matrix is generated in a Gaussian smoothing manner, which can avoid the problem of hard seam transition; and is extended to 3 channels, and pixel fusion is performed with the two images; the fusion formula is:

[0124] , wherein , is a coordinate variable of an image pixel, is the result of an inflation operation on the overlapping region mask M, is a pixel value of the first input image, is a pixel value of the second input image, and k is a channel index; when =1, the pixel of the second input image is completely retained, when , the pixel of the first input image is completely retained, and when , linear weighted fusion is implemented; in this way, a transition natural stitching image can be obtained.

[0125] S52: the transition of the first input image and the second input image is implemented through the smooth weight matrix , and the overlapping region refers to a region where the first input image and the second input image both have valid pixels at the same coordinate position, and the mask of the overlapping region is calculated according to the formula:

[0126] , wherein is the first input image, is the second input image, indicates that the first input image has a valid pixel at . Through the mask, the overlapping part of the two images is accurately located, so that the weighted fusion strategy is applied in this region to achieve natural transition.

[0127] Reference Figure 3As shown, in some embodiments of the present application, in order to ensure the quality of the final output image after image stitching is completed, the method further comprises:

[0128] S501: The root mean square error (RMSE) is used to measure the registration accuracy, which is defined as follows: , wherein, is the number of inliers; is the inlier coordinate of the first input image; is the inlier coordinate of the second input image after transformation.

[0129] Wherein, RMSE (root mean square error) is the geometric alignment accuracy, which measures the coordinate error of the inliers after transformation, and reflects the geometric alignment accuracy of the second input image and the first input image. The smaller the RMSE, the more accurate the registration.

[0130] S502: The stitching quality is quantified by SSIM: , wherein is the window pixel of the stitched image, is the corresponding window pixel of the first input image; is the mean value (luminance metric) of the window pixel ; ; is the standard deviation (contrast metric) of the window pixel ; is the standard deviation of the window pixel ; is the covariance (structure metric) of the window pixel and ; , , is a constant to avoid division by zero; is the total number of window pixels.

[0131] Wherein, SSIM (structural similarity index) is the image structure consistency, which evaluates the consistency of the stitched image and the first input image from three dimensions of luminance, contrast and structure, and the range is [0, 1] (1 represents complete consistency).

[0132] S503: The stitching quality is quantified by PSNR: , wherein is the image bit depth; MSE is the mean square error, which only calculates the pixel difference of the overlapping area, and the formula is:

[0133] , is the stitching overlapping area, is the first input image overlapping area.

[0134] ​Wherein, PSNR (peak signal to noise ratio) is the pixel difference degree, which measures the pixel gray difference between the stitching image and the first input image, and the larger the value is, the smaller the difference is (the better the quality is).

[0135] S504: Quantify the stitching quality by EdgeSmooth:

[0136] Wherein , is the gray image of , is the edge pixel set of the overlapping area, calculates the standard deviation of the edge pixel gray difference.

[0137] Wherein, EdgeSmooth (edge transition smoothness) is the seam obviousness, which measures the gray transition smoothness of the edge of the overlapping area, and the smaller the standard deviation is, the less obvious the seam is.

[0138] In the image stitching process, black edges may appear in the stitching result due to geometric transformation and boundary filling. Therefore, after the stitching is completed, the image is detected and cropped for black edges, and only the area containing valid pixels is retained, so as to further optimize the final stitching effect.

[0139] In some embodiments of the present application, for the above process of completing image stitching and quality optimization, the method further includes an output mode of the stitching result, so as to uniformly manage and store the stitching effects of different algorithms. Specifically, the stitching result is output in the form of a cell array.

[0140] The stitching result is denoted as:

[0141] Wherein is a 1x4 cell array, and each element in the array corresponds to the output result image of one stitching algorithm. For any one stitching result , its pixel value satisfies the following condition:

[0142] Wherein is the value range, indicating that the pixel value is represented by 8-bit unsigned integer (uint8).

[0143] By outputting in the form of a cell array, multiple groups of results obtained by different stitching strategies (such as weighted average fusion, pyramid fusion, multi-band fusion, etc.) can be saved at the same time, which is convenient for subsequent comparison of the stitching effects of different methods.

[0144] Therefore, as shown in Figures 4 to 6 , Figure 4The feature point matching diagram of the method in 4 is shown, Figure 5 The splicing result diagram of the four methods is shown, Figure 6 The evaluation index diagram of the four methods is shown. It can be seen that the evaluation index value of the method (FUSION) of the application is better than that of the existing optimization method (SIFT, ORB and SUFT).

[0145] Reference Figure 7 As shown in the drawings, in some embodiments of the application, the application also provides a multi-feature fusion and adaptive optimization SIFT image splicing system using the above method, comprising:

[0146] The data processing module 201 is used for acquiring image data and performing data preprocessing on the acquired image data to generate a gray image;

[0147] The feature matching module 202 is used for constructing a scale space and detecting local extreme points in the gray image by using the SIFT feature extraction method, generating a descriptor, and then performing SIFT feature matching optimization; by using the nearest neighbor ratio criterion and the matching threshold value to eliminate false matches and using a uniform sampling method to maintain the distribution balance of feature points in the gray image, the calculation amount is reduced;

[0148] The image registration module 203 is used for using adaptive threshold value to perform random sample consensus estimation in the feature matching point set, eliminating outliers and calculating a homography matrix, and realizing image registration;

[0149] The splicing optimization module 204 is used for realizing image splicing in the overlapping area based on a weighted fusion strategy, avoiding the abrupt transition of the joint by combining a smoothing weight, and performing black border cropping after splicing to optimize the splicing result.

[0150] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0151] Although the embodiments of the application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.

Claims

1. A multi-feature fusion and adaptive optimization SIFT image stitching method, characterized in that, Includes the following steps: S1: Perform data preprocessing on the acquired image data to generate a grayscale image; S2: Construct a scale space using the SIFT feature extraction method and detect local extrema in the grayscale image to generate descriptors, and then perform SIFT feature matching optimization; S3: False matches are eliminated using the nearest neighbor ratio criterion and matching threshold, and a uniform sampling method is used to maintain the balanced distribution of feature points in the grayscale image, thereby reducing the computational load. The specific steps are as follows: S31: Euclidean distance is used as the distance metric for the descriptor, and the calculation formula is as follows: wherein, is a descriptor dimension, are two descriptors, is the Euclidean distance between two descriptors, is a descriptor dimension index; S32: Feature matching using the nearest neighbor ratio criterion: for descriptors Find the two closest descriptors in the first input image. and ,like Then keep and The match, where for Indicates the first input image and the descriptor The corresponding feature point coordinates, yes If the corresponding feature point is selected, it is considered a mismatch and discarded. wherein the matching constraint is set wherein, to describe the sub Euclidean distance between the first input image descriptor Euclidean distance between the first input image descriptor Only points with distance less than 50 are allowed to be kept; S33: The grayscale image is divided into grids of a preset size using uniform sampling. The coordinate range formula for the grid is: wherein for each grid, points whose coordinates satisfy and are screened out and saved according to a set sampling strategy to provide data for subsequent registration; wherein, is the left boundary column coordinate of the i-th column grid, is the right boundary column coordinate of the i-th column grid, is the upper boundary row coordinate of the j-th row grid, is the lower boundary row coordinate of the j-th row grid, is the number of grid divisions; S4: Use an adaptive threshold to perform random sampling consistency estimation in the feature matching point set, remove outliers and obtain the homography matrix to achieve image registration; S5: Image stitching is performed in the overlapping area based on a weighted fusion strategy, combined with smoothing weights to avoid abrupt seam transitions. After stitching, black borders are cropped to optimize the stitching result. The specific steps are as follows: S51: Generate smoothing weights for overlapping regions between images. : Using standard deviation The Gaussian kernel smooths the weights, avoiding abrupt transitions; thus, the Gaussian kernel smooths the weights, preventing abrupt changes. Expanded to 3 channels, the fusion formula is: wherein , is a coordinate variable of an image pixel, is a result of an expansion operation on the overlap region mask, = 1 completely retains the pixel of the second input image, = 0 completely retains the pixel of the first input image, 0 < 1 linear weighting, realizing a smooth transition; k is a channel index, is a pixel value of the first input image, is a pixel value of the second input image; S52: by a smooth weight matrix The transition of the first input image and the second input image is realized, the overlapping region is the region where the first input image and the second input image have pixels, and the mask overlapMask is calculated according to the following formula: wherein is a first input image, is a second input image, denotes that the first input image has valid pixels at . 2.The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 1, characterized in that, In step S1, the method includes: S11: reading at least two RGB input images from the collected image data and defining them as wherein , is the number of image pixel rows, is the number of image pixel columns, is the data type of the image pixel values is an integer; S12: Perform grayscale processing. , For the first i line, number j Column, pixel value of the first channel, For the first i line, number j Column, second channel pixel value, For the first i line, number j Column, 3rd channel pixel value, To round down; S13: The generated grayscale image is .

3. The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 2, characterized in that, In step S2, the method includes: S21: a method of Gaussian pyramid is used to construct the scale space of the image, to ensure that the feature points can be detected under different scales, wherein the scale space is defined as , and the calculation formula is: , is the input gray image, is a two-dimensional convolution, is a Gaussian kernel function, , is a scale parameter; S22: Calculate the Gaussian difference image between adjacent scales: wherein is a scale factor of adjacent scales, is a scale space function; and then for each pixel in , compared with 8 pixels of the same scale and 18 pixels of the two adjacent scales above and below, if it is a local extreme value, it is considered as a candidate feature point; S23: For each detected candidate feature point, extract a 16×16 neighborhood region centered on that feature point, and then calculate the gradient magnitude and direction for each pixel in the neighborhood: ， , is a gradient magnitude of the pixel point, is a pixel value of the scale space image L at coordinate (x+1, y), is a pixel value of the scale space image L at coordinate (x-1, y), is a pixel value of the scale space image L at coordinate (x, y+1), is a pixel value of the scale space image L at coordinate (x, y-1), is a gradient direction of the pixel point; S24: Construct a gradient direction histogram, count the gradient directions in the neighborhood, and select the peak direction in the histogram as the main direction of the feature point; to achieve scale invariance, divide the 16×16 neighborhood into 4×4 sub-blocks, count the gradient histograms of 8 directions in each sub-block, and finally generate a 4×4×8=128-dimensional descriptor.

4. The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 1, characterized in that, In step S4, the method includes: S41: let a point in the second input image be , homogeneous coordinates ; let the corresponding point in the first input image be , homogeneous coordinates , then: where is a 3x3 homography matrix, , expanding its second equation gives , ; S42: Find the optimal homography matrix by using RANSAC registration method .

5. The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 4, characterized in that, In step S42, the method includes: S421: define inlier threshold wherein , is the Euclidean distance for all matching point pairs, , is the standard deviation of the distances; S422: Randomly select 4 points from the matched points, substitute the 4 points into the projection transformation formula to generate 8 linear equations, and calculate the coordinates of the nth point , the calculation process is the formula: ； S423: Solve by singular value decomposition .

6. The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 5, characterized in that, In step S42, the method further comprises: calculating, for all matching points, Transformed prediction points Transformed prediction points If If the current number of inliers is greater than the historical maximum, update the optimal If the current number of inliers is greater than the historical maximum, update the optimal If the current number of inliers is greater than the historical maximum, update the optimal If the current number of inliers is greater than the historical maximum, update the optimal 7. The multi-feature fusion and adaptive optimization SIFT image stitching method according to claim 1, characterized in that, After the image stitching is completed, the method further includes: S501: Employs root mean square error to quantify stitching quality. wherein, is the number of inliers; is the inlier coordinate of the first input image; is the inlier coordinate of the second input image; is the transformed predicted coordinate of the inlier. S502: Employing a structural similarity index to quantify stitching quality. wherein is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; is the mean of the window pixels of the first input image; , , is a constant to avoid division by zero; is the total number of window pixels; S503: Quantify the stitching quality by peak signal to noise ratio wherein is the image bit depth; MSE is the mean square error, only calculate the pixel difference of the overlapping area, the formula is , is the overlapping area of the stitching, is the overlapping area of the first input image; S504: Quantify the splicing quality by edge transition smoothness: wherein , is a gray image, is a set of edge pixels of the overlapping area, calculates the standard deviation of the gray difference of the edge pixels.

8. A multi-feature fusion and adaptive optimization SIFT image stitching system using the method of any one of claims 1-7, characterized in that, include: The data processing module is used to acquire image data and perform data preprocessing on the acquired image data to generate grayscale images; The feature matching module is used to construct a scale space using the SIFT feature extraction method and detect local extrema points in the grayscale image, generate descriptors, and then perform SIFT feature matching optimization; it removes mismatches by using the nearest neighbor ratio criterion and matching threshold, and maintains the distribution balance of feature points in the grayscale image by using uniform sampling, thereby reducing the amount of computation; The image registration module is used to perform random sampling consistency estimation on the feature matching point set using an adaptive threshold, remove outliers and obtain the homography matrix to achieve image registration. The stitching optimization module is used to stitch images in overlapping areas based on a weighted fusion strategy and combine smoothing weights to avoid abrupt seam transitions. After stitching, black borders are cropped to optimize the stitching result.

Images