Unmanned aerial vehicle image splicing method based on local distortion and seam guidance

Abstract

The invention discloses an unmanned aerial vehicle image splicing method based on local distortion and seam guidance, and relates to the technical field of computer image splicing. The technical problem that an image splicing result generates artifacts due to the fact that parallax exists between input images in image splicing and warping characteristics of non-overlapping areas of the images are not fully considered is solved. According to the method, image warping is guided by using a grid, labels are created among pixels in an overlapping region, and a logic function is designed to optimize the minimization of a label energy function; meanwhile, searching an optimal seam in the overlapping region by adopting a seam cutting method; the method focuses on an unmanned aerial vehicle remote sensing image, and solves artifacts caused by parallax from the image warping characteristic; according to the method, the target image is firstly subjected to grid warping processing, then the optimal splicing seam is found by using the seam cutting method and is fused with the reference image, and finally accurate alignment is realized, so that the quality of the spliced image is effectively improved, and a better scheme is provided for unmanned aerial vehicle image splicing.

Description

Technical Field

[0001] The present invention belongs to the field of computer image stitching, and relates to a parallax-based drone image stitching technology, and specifically to a drone image stitching method based on local distortion and seam guidance. Background Art

[0002] As an important technology in the field of computer vision, computer image stitching technology is widely used in remote sensing image analysis, medical imaging, virtual reality and other fields. It aims to integrate a set of images with overlapping areas into a seamless panoramic image or high-resolution image. In the image stitching process, it is difficult to obtain sufficient and correct matching points for low-texture or repeated-texture areas using the SIFT+RANSAC algorithm, resulting in insufficient matching points in overlapping areas, which in turn leads to incorrect estimation of the image warping model. The correctness and number of matching points between images directly affect the final stitching effect.

[0003] When the parallax between images is large, the stitching results obtained by traditional image stitching technology will have more significant misalignment artifacts and perspective distortion. In this case, the simple image superposition method cannot effectively correct the pixel information of the misaligned position caused by inaccurate alignment. To solve this problem, the seam line fusion algorithm is used to cleverly avoid some pixels with severe misalignment, which optimizes the stitching effect to a certain extent. However, the seam fusion algorithm depends on the selection and design of the energy function. If the energy function is not selected properly, it will not only fail to effectively reduce the artifacts and distortion in the stitching, but also affect the final stitching effect.

[0004] The present invention proposes a UAV image stitching method based on local distortion and seam guidance to solve the above technical problems. Summary of the invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art; to this end, the present invention proposes a drone image stitching method based on local distortion and seam guidance, which is used to solve the problem that in the image stitching process, there is parallax between input images and the warping characteristics of non-overlapping areas of the images are not fully considered, resulting in artifacts in the image stitching results.

[0006] To achieve the above object, the present invention provides a UAV image stitching method based on local distortion and seam guidance, comprising:

[0007] Step 1: Obtain image data set and add matching point pairs;

[0008] Obtain a UAV remote sensing image dataset, use SIFT to detect and match feature points, use projection invariance to explore coplanar regions, and increase matching point pairs; among them, the coplanar region is constructed by detecting adjacent lines in the image;

[0009] Step 2: Grid division and calculation;

[0010] Divide the target image into grids and set homography weights Calculate the local homography matrix h for each grid * , obtaining a warped image according to the local homography matrix, wherein the warped image includes warped images of the target image and the reference image;

[0011] Step 3: Set labels and obtain initial seams;

[0012] Set the label set L for the overlapping area P of the target image and the reference image, and assign a label l to each pixel p in the seam p , the energy function E(l) is minimized by the data term D p (l p ) and the smoothing term S p,q (l p ,l q ) to obtain the initial seam S 0 ; Among them, data item D p (l p ) is the pixel p corresponding to label l p The cost of the smoothing term S p,q (l p ,l q ) is a pair of adjacent pixels (p, q) corresponding to a pair of labels (l p ,l q ) costs;

[0013] Step 4: Evaluate and adjust the seams;

[0014] Evaluate the initial seam S 0 For each pixel, calculate the SSIM error Q(p) of the corresponding block of each pixel point, and use the Ostu algorithm to get τ based on the SSIM error Q(p) obtained above. If the error is greater than the threshold τ, it is an unreasonable seam. Extract the local block of misaligned pixels, and adjust the local block of misaligned pixels according to the SIFT flow between pixels in the local block of misaligned pixels to obtain the adjusted warped image and the new local seam s i , the initial seam S 0 With new local seams i Combine to get the final seam S * ;

[0015] Step 5: Generate the final stitched image;

[0016] The adjusted warped image and the final seam S * The final stitched image is obtained through gradient fusion to complete the alignment of drone images.

[0017] Preferably, in step 1, using projection invariance to explore coplanar areas and increase matching point pairs includes:

[0018] Get image data, extract image feature points and adjacent line features, divide the image into multiple coplanar sub-regions, use projection invariant parameters CN to construct line point invariance, use five points to build a triangle, and set equal intersection points on each edge, calculate sub-region CN similarity, and obtain continuous matching point pairs.

[0019] Preferably, in step 2, the homography weight is set include:

[0020] Divide the target image into multiple grids and calculate the homography weights using the weight formula The formula is as follows:

[0021]

[0022] Among them, x * is the target image pixel, x i Match {x i ,x' i} half coordinate, x' i represents the point in the reference image, σ is the scale parameter, and γ is used to offset the weight;

[0023] The diag function creates a diagonal matrix, the weight matrix W, based on the given homography weight vector. * , the formula is as follows:

[0024]

[0025] Where N is the number of matching points.

[0026] Preferably, in step 2, the local homography matrix h of each grid is calculated by the formula * , the formula is as follows:

[0027]

[0028] Among them, W * is the weight matrix, matrix N is the number of point matches, h is a vector, and the warped images of the target image and the reference image are obtained according to the local homography matrix.

[0029] Preferably, in step 3, the energy function E(l) is minimized by the data item D p (l p ) and the smoothing term S p,q (l p ,l q ) composition, including:

[0030] The target image I 0 and reference image I 1 Set the label set L for the overlapping area P and assign a label l to each pixel p∈P in the seam p ∈L, the label l is calculated by minimizing the energy function formula, the formula is as follows:

[0031]

[0032] Where N is the neighborhood system of the pixel, Data item D p (l p ) is to label l p The cost assigned to pixel p, the smoothing term S p,q (l p ,l q ) is a pair of labels (l p ,l q ) is assigned to a pair of adjacent pixels (p, q), W p,q is the significance weight.

[0033] Preferably, the initial seam S obtained in step 3 is 0 ,include:

[0034] The average pixel significance of the overlapping area is calculated by the weight formula, which is as follows:

[0035]

[0036] Where w( p ) and w(q) are the saliency of pixel p and pixel q in the overlapping region, respectively. Indicates that p or q is located at the common boundary of the canvas and p

[0037] The initial spliced ​​image and initial seam S are obtained by gradient fusion 0 .

[0038] Preferably, in step 4, each pixel of the initial seam is evaluated, and the SSIM error of the block corresponding to each pixel is calculated by a formula, which is as follows:

[0039]

[0040] Among them, P 0 (p i ),P 1 (p i ) is p i The local block centered on the image is obtained by using the Ostu algorithm based on the SSIM error Q(p) obtained above.i )≥τ is an unreasonable seam, and unaligned pixel local blocks are extracted in the target image and the reference image respectively.

[0041] Preferably, in step 4, the unaligned pixel local block is adjusted according to the SIFT flow between pixels in the unaligned pixel local block. Each pixel p in the local block j , there exists a shift vector V i (p j ) satisfies the following formula:

[0042]

[0043] Here, i refers to the misaligned pixel local block, and j refers to the pixel in the misaligned pixel local block.

[0044] Preferably, in step 4, when the local block When the boundary pixel has a large offset vector, the offset vector flow is smoothly modified to f(V i (p j )), the formula is as follows:

[0045]

[0046] Among them, β is a parameter used to control the rate at which the function f changes;

[0047] Compute the new local seam s by aligning the patches i , merge the initial seam with the new local seam to obtain the final seam S * .

[0048] Preferably, in step 5, obtaining the final stitched image by gradient fusion according to the adjusted warping map and the final seam includes:

[0049] By adjusting the misaligned blocks, the warp map and the final seam S are adjusted using the SIFT vector flow * , gradient fusion technology is used to smoothly transition the boundaries between images.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention uses grids to guide the warping of images, designs a logic function to optimize the label energy function minimization problem, and uses the seam cutting method to search for seams in overlapping areas. Combining these two methods not only improves the accuracy and quality of image stitching, but also can effectively deal with images in complex scenes; by dividing the target image into several grids, the local homography matrix of each grid is calculated according to the homography weight, and the image warping is precisely controlled. The accuracy of image stitching is improved, ensuring a smooth and natural transition between images.

[0052] 2. The present invention performs grid warping on the target image, finds the optimal seam through the seam cutting method, and then merges it with the reference image to obtain the final stitching image, thereby achieving precise alignment of drone images.

[0053] 3. The present invention obtains the initial seam by minimizing the energy function, and evaluates each pixel of the initial seam. When the SSIM error of the corresponding block exceeds the set threshold, the seam is considered unreasonable. The local blocks of these misaligned pixels are extracted, and the SIFT vector flow is used to adjust these local blocks to obtain the adjusted warping map and the updated local seam. The adjusted local seam is merged with the initial seam to obtain the final seam. This process not only improves the accuracy of the seam, but also enhances the overall quality of image stitching, achieving a smoother and more natural image stitching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is a work flow chart of the present invention;

[0056] Figure 2 The figure is a flow chart of an image stitching method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1-Figure 2, an embodiment of the present invention provides a drone image stitching method based on local distortion and seam guidance, comprising:

[0059] Step 1: Obtain image dataset;

[0060] A UAV remote sensing image dataset is obtained, and SIFT is used to detect and match feature points. Projection invariance is used to explore coplanar regions and increase matching point pairs. The coplanar regions are constructed by detecting adjacent lines in the image.

[0061] Exemplarily, the drone image dataset is processed with single precision and grayscale, feature points and adjacent line features in the image are extracted, the projection invariant parameter CN is used to construct line-point invariance, continuous matching point pairs are obtained, and the random sample consistency algorithm is used to effectively eliminate outliers and false matches to obtain the final matching point pairs.

[0062] Step 2: Grid division and calculation;

[0063] Divide the target image into grids and set homography weights Calculate the local homography matrix h for each grid * , obtaining a warped image according to the local homography matrix, wherein the warped image includes warped images of the target image and the reference image;

[0064] For example, the target image is divided into several grids (C1*C2), and the homography weight is determined according to the distance between each pixel and the upper left corner of the pixel grid by the weight formula The formula is as follows:

[0065]

[0066] Among them, x * is the target image pixel, x i Match {x i ,x' i} half coordinate, x' i represents the point in the reference image, σ is the scale parameter, and γ is used to offset the weight;

[0067] The diag function creates a diagonal matrix, the weight matrix W, based on the given homography weight vector. * , the formula is as follows:

[0068]

[0069] Where N is the number of matching points.

[0070] The local homography matrix h of each grid is calculated by the formula * , the formula is as follows:

[0071]

[0072] Among them, W * is the weight matrix, matrix N is the number of point matches, h is a vector, and the constraint is ||h|| = 1. The warped images of the target image and the reference image are obtained according to the local homography matrix.

[0073] Step 3: Set labels and obtain initial seams;

[0074] Set the label set L for the overlapping area P of the target image and the reference image, and assign a label l to each pixel p in the seam p , minimized by the data item D p (l p ) and the smoothing term S p,q (l p ,l q ) to form the energy function E(l) and obtain the initial seam S 0 ;

[0075] Exemplarily, set P to be the target image I 0 and reference image I 1 The overlapping area is set, and the label set L = {0, 1} is set for each pixel in the area; where 0 represents I 0 , 1 represents I 1 ; Assign a label l to each pixel p∈P of the seam p ∈L, the label l is calculated by minimizing the energy function formula E(l); the formula is as follows:

[0076]

[0077] Where N is the neighborhood system of the pixel, Data item D p (l p ) is to label l p The cost assigned to pixel p, the smoothing term S p,q (l p ,l q ) is a pair of labels (l p ,l q ) is assigned to a pair of adjacent pixels (p, q), W p,q is the significance weight.

[0078] The initial seam S is obtained in step 3 0 ,include:

[0079] The average pixel significance of the overlapping area is calculated by the weight formula, which is as follows:

[0080]

[0081] Where w(p) and w(q) are the saliency of pixel p and pixel q in the overlapping region, respectively. Indicates that p or q is located at the common boundary of the canvas and p

[0082] The initial spliced ​​image and initial seam S are obtained by gradient fusion 0 .

[0083] Step 4: Evaluate and adjust the seams;

[0084] Evaluate each pixel of the initial seam and calculate the SSIM error of the block corresponding to each pixel using the formula:

[0085]

[0086] Among them, P 0 (p i ),P 1 (p i ) is p i The local block centered on S is considered to be 0 is a reasonable seam; where k = 1.5; otherwise, the threshold τ is calculated by the Ostu method based on the SSIM error Q(p) obtained above, and Q(p i )≥τ is considered as misalignment; extract the local blocks of misaligned pixels in the target image and the reference image respectively;

[0087] The unaligned pixel local blocks are adjusted according to the SIFT flow between pixels in the unaligned pixel local blocks. Each pixel p in the local block j , there exists a shift vector V i (p j ) satisfies the following formula:

[0088]

[0089] Here, i refers to the misaligned pixel local block, and j refers to the pixel in the misaligned pixel local block.

[0090] When the local block When the boundary pixel has a large offset vector, the offset vector flow is smoothly modified to f(V i (p j )), the formula is as follows:

[0091]

[0092] Among them, β is a parameter used to control the rate at which the function f changes;

[0093] The patch size is (21×21), β=8; the new local seam s is calculated by aligning the patches i , merge the initial seam with the new local seam to get the final seam S * The new local seam s is calculated by aligning the patches i , merge the initial seam with the new local seam to obtain the final seam S * .

[0094] Step 5: Generate the final stitched image;

[0095] According to the adjusted warped image and the final seam S * ,The final stitched image is obtained through gradient fusion, and the alignment of the UAV images is completed.

[0096] Exemplarily, the adjusted target image, the warped image of the reference image and the final stitching seam are extracted, and the gradient domain fusion algorithm is used to smoothly transition the boundaries between the images to obtain a seamless and high-quality final stitching image to achieve precise alignment of drone images.

[0097] Working principle of the present invention:

[0098] Obtain a drone image dataset, and use projection invariance to divide the image into multiple coplanar sub-regions; Use SIFT to detect and match feature points from the image dataset, divide the target image into several grids, calculate the local homography matrix of each grid, and obtain the warped image of the target image and the reference image based on the local homography matrix; Set a label set for the overlapping area P of the target image and the reference image, assign a label to each pixel p of the seam, obtain the label l by minimizing the energy function, and calculate the average pixel significance of the overlapping area; Obtain the initial stitched image and the initial seam through gradient fusion. Evaluate each pixel of the initial seam, calculate the SSIM error of the corresponding block of each pixel point, and when the error exceeds the set threshold τ, the seam is unreasonable; Extract the local blocks of misaligned pixels, use SIFT vector flow to adjust the misaligned blocks, obtain the adjusted warped map and updated local seam, and merge the initial seam with the adjusted local seam to obtain the final seam. According to the warped map and final seam S adjusted using SIFT vector flow * ,The gradient fusion technology is used to smoothly transition the boundaries between images to obtain the final stitched image and achieve accurate alignment of UAV images.

[0099] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A UAV image stitching method based on local distortion and seam guidance, characterized in that: include: Step 1: Obtain image data set and add matching point pairs; Obtain a UAV remote sensing image dataset, use SIFT to detect and match feature points, use projection invariance to explore coplanar regions, and increase matching point pairs; among them, the coplanar region is constructed by detecting adjacent lines in the image; Step 2: Grid division and calculation; Divide the target image into grids and set homography weights Calculate the local homography matrix h for each grid * , obtaining a warped image according to the local homography matrix, wherein the warped image includes warped images of the target image and the reference image; Step 3: Set labels and obtain initial seams; Set the label set L for the overlapping area P of the target image and the reference image, and assign a label l to each pixel p in the seam p , the energy function E(l) is minimized by the data term D p (l p ) and the smoothing term S p,q (l p ,l q ) to obtain the initial seam S0; where the data item D p (l p ) is the pixel p corresponding to label l p The cost of the smoothing term S p,q (l p ,l q ) is a pair of adjacent pixels (p, q) corresponding to a pair of labels (l p ,l q ) costs; Step 4: Evaluate and adjust the seams; Evaluate each pixel of the initial seam S0, calculate the SSIM error Q(p) of each pixel corresponding to the block, and use the Ostu algorithm to obtain τ based on the SSIM error Q(p) obtained above. If the error is greater than the threshold τ, it is an unreasonable seam. Extract the local block of misaligned pixels, and adjust the local block of misaligned pixels according to the SIFT flow between pixels in the local block of misaligned pixels to obtain the adjusted warped image and the new local seam s i , the initial seam S0 is combined with the new local seam s i Combine to get the final seam S * ; Step 5: Generate the final stitched image; The adjusted warped image and the final seam S * The final stitched image is obtained through gradient fusion to complete the alignment of drone images.

2. The method for UAV image stitching based on local distortion and seam guidance according to claim 1, characterized in that: In the step 1, the coplanar area is explored using projection invariance to increase matching point pairs, including: Get image data, extract image feature points and adjacent line features, divide the image into multiple coplanar sub-regions, use projection invariant parameters CN to construct line point invariance, use five points to build a triangle, and set equal intersection points on each edge, calculate sub-region CN similarity, and obtain continuous matching point pairs.

3. The method for UAV image stitching based on local distortion and seam guidance according to claim 1, characterized in that: In step 2, set the homography weight include: Divide the target image into multiple grids and calculate the homography weights using the weight formula The formula is as follows: Among them, x * is the target image pixel, x i Match {x i ,x' i } half coordinate, x' i represents the point in the reference image, σ is the scale parameter, and γ is used to offset the weight; The diag function creates a diagonal matrix, the weight matrix W, based on the given homography weight vector. * , the formula is as follows: Where N is the number of matching points.

4. The method for stitching drone images based on local distortion and seam guidance according to claim 3, characterized in that: In step 2, the local homography matrix h of each grid is calculated by the formula: * , the formula is as follows: Among them, W * is the weight matrix, matrix N is the number of point matches, h is a vector, and the warped images of the target image and the reference image are obtained according to the local homography matrix.

5. The method for UAV image stitching based on local distortion and seam guidance according to claim 1, characterized in that: In step 3, the energy function E(l) is minimized by the data item D p (l p ) and the smoothing term S p,q (l p ,l q ) composition, including: Set the label set L for the overlapping area P of the target image I0 and the reference image I1, and assign a label l to each pixel p∈P of the seam p ∈L, the label l is calculated by minimizing the energy function formula, the formula is as follows: Where N is the neighborhood system of the pixel, Data item D p (l p ) is to label l p The cost assigned to pixel p, the smoothing term S p,q (l p ,l q ) is a pair of labels (l p ,l q ) is assigned to a pair of adjacent pixels (p, q), W p,q is the significance weight.

6. The method for stitching drone images based on local distortion and seam guidance according to claim 5, characterized in that: The initial seam S0 is obtained in step 3, including: The average pixel significance of the overlapping area is calculated by the weight formula, which is as follows: Where w(p) and w(q) are the saliency of pixel p and pixel q in the overlapping region, respectively. Indicates that p or q is located at the common boundary of the canvas and p The initial stitching image and the initial seam S0 are obtained by gradient fusion.

7. The method for UAV image stitching based on local distortion and seam guidance according to claim 1, characterized in that: In step 4, each pixel of the initial seam is evaluated, and the SSIM error of the block corresponding to each pixel is calculated by the formula, which is as follows: Among them, P0(p i ),P1(p i ) is p i The local block centered on the image is obtained by using the Ostu algorithm based on the SSIM error Q(p) obtained above. i )≥τ is an unreasonable seam, and unaligned pixel local blocks are extracted in the target image and the reference image respectively.

8. The method for UAV image stitching based on local distortion and seam guidance according to claim 7, characterized in that: In the step 4, the unaligned pixel local block is adjusted according to the SIFT flow between pixels in the unaligned pixel local block. Each pixel p in the local block j , there exists a shift vector V i (p j ) satisfies the following formula: Here, i refers to the misaligned pixel local block, and j refers to the pixel in the misaligned pixel local block.

9. The method for UAV image stitching based on local distortion and seam guidance according to claim 8, characterized in that: In step 4, when the local block When the boundary pixel has a large offset vector, the offset vector flow is smoothly modified to f(V i (p j )), the formula is as follows: Among them, β is a parameter used to control the rate at which the function f changes; Compute the new local seam s by aligning the patches i , merge the initial seam with the new local seam to obtain the final seam S * .

10. The method for UAV image stitching based on local distortion and seam guidance according to claim 9, characterized in that: In the step 5, the final stitched image is obtained by gradient fusion according to the adjusted warping map and the final seam, including: By adjusting the misaligned blocks, the warp map and the final seam S are adjusted using the SIFT vector flow * , gradient fusion technology is used to smoothly transition the boundaries between images.

Images