An aircraft skin-oriented weak-texture curved surface image stitching method and device

Abstract

This invention discloses a method and apparatus for stitching weakly textured curved surface images for aircraft skin. The method includes: correcting the acquired original image using cylindrical back projection based on the position information of the camera and the aircraft model, and using nearest neighbor interpolation to fill in missing pixels in the image; calculating the overlapping area between adjacent images using the camera position information and image size to obtain a feature mask; extracting feature points using the LOFTR algorithm, and deleting feature point pairs in non-overlapping areas based on the feature mask; dividing the image into a grid according to the idea of ​​image segmentation, and deleting redundant feature points in densely populated areas; transforming the image using the RANSAC algorithm to select a similarity transformation matrix with the optimal number of interior points, and using fade-in / fade-out weighted average to eliminate seams between images, achieving seamless stitching. The apparatus includes a processor and a memory. This invention solves the problem of poor stitching results caused by the small number and uneven distribution of detected feature points in aircraft skin images.

Description

Technical Field

[0001] This invention relates to the field of image stitching, and more particularly to a method and apparatus for stitching weakly textured curved surface images for aircraft skin. Background Technology

[0002] Aircraft skin refers to the structural components that surround the aircraft's frame, serving functions such as protecting the aircraft structure, controlling airflow, and improving fuel efficiency. Therefore, airlines need to conduct regular inspections of the aircraft skin to improve aircraft performance and efficiency and extend its service life.

[0003] Currently, aircraft skin inspection mainly involves personnel handheld cameras taking pictures of the aircraft skin, followed by image analysis. This manual inspection method is inefficient. With the development of automated inspection technology, using Automated Guided Vehicles (AGVs) equipped with cameras for automated image acquisition can greatly improve efficiency. However, due to the limitations of camera angles, a single image has a small field of view, and the acquired images cannot display more complete information about the aircraft skin, resulting in poor analysis results. To obtain high-resolution aircraft skin images, the acquired images can be stitched together, thereby achieving a wider range of perception of the aircraft skin.

[0004] Many mature image stitching algorithms, such as APAP (Aspect-As-Positive Projection Image Stitching), Auto-Stitch (Automatic Panoramic Image Stitching Using Invariant Features), and SPHP (Shape-Preserving Semi-Projection Transform in Image Stitching), utilize traditional feature extractors, such as SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust), and ORB (Directional Fast and Rotation BRIEF). These traditional feature extraction algorithms can detect more feature points at image edges, corners, or areas with rich texture, thus completing image stitching. However, since aircraft skin images are typically weakly textured, applying these feature extraction algorithms presents two problems: ① Traditional feature extraction algorithms are usually based on image gradient information, but weakly textured images have fewer corners and no significant boundaries, resulting in less gradient information and fewer detected feature points, making image registration impossible; ② Feature descriptors encode information about the area surrounding local feature points. Feature points in weakly textured areas contain less information and are easily affected by noise. Therefore, traditional image stitching algorithms are difficult to apply to aircraft skin image stitching.

[0005] With the development of deep learning, some deep learning-based feature extraction algorithms, such as SuperPoint7 and NC-Net8, still perform poorly in weakly textured regions. The LoFTR algorithm proposed by Sun9 et al. uses attention mechanisms and positional encoding to generate dense matching in weakly textured regions, but it suffers from the following problems in aircraft skin stitching: ① This algorithm is designed for images with a large overlap area, while aircraft skin images taken by AGVs have a small overlap area, resulting in many mismatched feature point pairs in non-overlapping areas; ② This algorithm generates relatively more matching point pairs in areas with significant texture, but the uneven distribution of texture on aircraft skin causes the generated feature points to be mainly concentrated in areas with significant texture, affecting image registration. Summary of the Invention

[0006] This invention provides a method and apparatus for stitching weakly textured surface images of aircraft skin. This invention solves the problem of poor stitching results caused by a small number of detected feature points and their uneven distribution in aircraft skin images. Details are described below:

[0007] A first aspect is a method for stitching weakly textured surface images for aircraft skin, the method comprising:

[0008] Based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original image, and nearest neighbor interpolation is used to fill in the missing pixels in the image.

[0009] Using camera position information and image size, the overlapping area between adjacent images is calculated to obtain a feature mask;

[0010] The LOFTR algorithm is used to extract feature points, and feature point pairs in non-overlapping regions are removed based on the feature mask.

[0011] The image is divided into grids based on the idea of ​​image segmentation, and redundant feature points in dense feature point areas are deleted;

[0012] The RANSAC algorithm is used to select a similarity transformation matrix with the optimal number of interior points to transform the image, and fade-in / fade-out weighted methods are used to eliminate seams between images, achieving seamless stitching.

[0013] Specifically, the process of correcting the acquired original image using cylindrical back projection and filling in missing pixels in the image using nearest neighbor interpolation involves:

[0014] Based on the camera parameters, the width and height of the acquired image are w and h, respectively, the camera's field of view is α, and the focal length is f. The mapping relationship between the acquired original image and the image after cylindrical back projection in the horizontal and vertical directions is as follows:

[0015]

[0016]

[0017] Where θ represents the angle of the original image pixel, (x ′ ,y ′ () represents the pixel coordinates of the original image. According to formulas (1) and (2), the pixel coordinates (x, y) after inverse cylindrical projection can be obtained:

[0018]

[0019] Nearest neighbor interpolation is used to fill in the missing pixels in the image and approximate the distorted areas of the corrected image, thus obtaining the image after distortion correction.

[0020] The pixel value at coordinate (a,b) is: f(a,b)=(f(a-1,b)+f(a+1,b)+f(a,b-1)+f(a,b+1)) / 4.

[0021] The step of using camera position information and image size to calculate the overlapping area between adjacent images yields the following feature mask:

[0022] The size of the overlapping area between the two images is determined based on the displacement distance and the size of the acquired images. By setting a mask, mismatched feature point pairs in the non-overlapping area are deleted, while matching point pairs in the overlapping area are retained.

[0023] Given that the width and height of the image acquired by the AGV are w and h respectively, the distance from the optical center to the aircraft is z, the horizontal displacement of the AGV between two adjacent images is a, and the width of the overlapping area between adjacent images is P. mask The camera focal length is f, and the physical length of the aircraft skin in the image is x, where l and m represent the physical plane and the imaging plane, respectively. The following correspondence is obtained.

[0024]

[0025]

[0026] According to P mask Obtain the overlapping area between adjacent graphics and set a feature mask.

[0027] Furthermore, the step of dividing the image into grids based on the idea of ​​image segmentation and deleting redundant feature points in densely populated areas specifically involves:

[0028] Based on the size of the aircraft skin image, the image is divided into several 20×20 pixel grids;

[0029] The confidence score (conf) and pixel coordinates of the matching point pairs are obtained using the LoFTR algorithm.

[0030] Based on the confidence level (conf), all feature points within each grid are filtered, and the three feature points with the highest confidence levels are retained.

[0031] Other feature point pairs are treated as redundant and deleted to obtain the final matching point pairs between images.

[0032] A second aspect is a weakly textured curved surface image stitching device for aircraft skin, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any of the first aspects.

[0033] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0034] The beneficial effects of the technical solution provided by this invention are:

[0035] 1. This invention utilizes camera position information and image size to calculate the overlapping area between adjacent images; then, based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original image, and linear interpolation is used to fill in the missing pixels in the image; then, the LoFTR algorithm is used to extract feature points, and the feature points in the non-overlapping area are masked.

[0036] 2. This invention divides the image into grids based on the idea of ​​image segmentation, and deletes redundant feature points in dense feature point areas, so that the distribution of feature points is more uniform, resulting in a similarity transformation matrix that can balance the whole, making the image registration effect more accurate.

[0037] 3. This invention uses feature masks and mesh division to filter feature points in aircraft skin images, obtaining a sufficient number of relatively evenly distributed feature point pairs, resulting in better image registration. This effectively solves the problem of stitching weak texture images of aircraft skin, and the stitching effect is higher than that of existing methods. Attached Figure Description

[0038] Figure 1 A flowchart of the aircraft skin image stitching system provided by the present invention;

[0039] Figure 2 A schematic diagram of cylindrical back projection of an aircraft skin image provided by the present invention;

[0040] (a) is a schematic diagram in the horizontal direction; (b) is a schematic diagram in the vertical direction.

[0041] Figure 3 This is a schematic diagram of the camera translation model provided by the present invention;

[0042] Figure 4 A schematic diagram of the AGV acquiring images provided by the present invention;

[0043] Figure 5 A comparative diagram of the feature detection results provided by the present invention;

[0044] Figure 6 Example diagram of the feature statistical results provided by the present invention;

[0045] Figure 7 This is a comparative schematic diagram of the splicing results provided by the present invention.

[0046] Among them, (a) is a schematic diagram of SIFT stitching effect; (b) is a schematic diagram of ORB stitching effect; (c) is a schematic diagram of SURF stitching effect; (d) is a schematic diagram of BRISK stitching effect; (e) is a schematic diagram of AKAZE stitching effect; (e) is a schematic diagram of stitching effect with feature mask added; and (g) is a schematic diagram of stitching effect of this method. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0048] To address the problems existing in the background technology, this invention proposes an improved LoFTR algorithm for stitching aircraft skin images, thereby solving the problem of poor stitching effect of aircraft skin images caused by a large number of mismatched point pairs and uneven distribution of feature points.

[0049] First, based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original images, and linear interpolation is used to fill in the missing pixels in the images. Next, using the camera position information and image size, the overlapping area between adjacent images is calculated. Then, the LoFTR algorithm is used to extract feature points, and the feature points in non-overlapping areas are masked. Then, the image is divided into grids according to the idea of ​​image block division, and redundant feature points in dense feature point areas are deleted to make the distribution of feature points more uniform. Then, the RANSAC algorithm is used to select the similarity transformation matrix with the optimal number of interior points to transform the image, making the image registration effect more accurate. Finally, fade-in and fade-out weighted fusion is used to eliminate the seams between images.

[0050] Example 1

[0051] A method for stitching weakly textured surface images for aircraft skin, comprising the following steps:

[0052] 101: Based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original image, and nearest neighbor interpolation is used to fill in the missing pixels in the image;

[0053] 102: Using camera position information and image size, calculate the overlapping area between adjacent images to obtain a feature mask;

[0054] 103: Use the LOFTR algorithm to extract feature points and remove feature point pairs in non-overlapping regions based on the feature mask;

[0055] 104: Divide the image into grids based on the idea of ​​image segmentation, and delete redundant feature points in areas with dense feature points;

[0056] 105: The RANSAC algorithm is used to select a similarity transformation matrix with the optimal number of interior points to transform the image. Gradual-in and gradual-out weighting is used to eliminate seams between images, achieving seamless stitching.

[0057] Specifically, step 104 involves dividing the image into grids based on the idea of ​​image segmentation and deleting redundant feature points in densely populated regions.

[0058] Based on the size of the aircraft skin image, the image is divided into several 20×20 pixel grids;

[0059] The confidence score (conf) and pixel coordinates of the matching point pairs are obtained using the LoFTR algorithm.

[0060] Based on the confidence level (conf), all feature points within each grid are filtered, and the three feature points with the highest confidence levels are retained.

[0061] Other feature point pairs are treated as redundant and deleted to obtain the final matching point pairs between images.

[0062] In summary, the embodiments of the present invention solve the problem of poor stitching effect caused by the small number and uneven distribution of feature points detected in aircraft skin images through the above steps 101-105.

[0063] Example 2

[0064] The scheme in Example 1 will be further described below with specific calculation formulas and examples:

[0065] Step 201: Based on the position information of the camera and the aircraft model, the original image is corrected using cylindrical back projection, and the missing pixels in the image are filled in using the nearest neighbor interpolation method.

[0066] Specifically, step 201 includes:

[0067] S11: Use cylindrical back projection to perform surface correction on the aircraft skin image, and project the original image cylindrically onto the tangent plane of the cylinder.

[0068] The main body of the aircraft fuselage is approximately cylindrical. The skin structure captured by the AGV is not on the same plane, resulting in visual distortion. To preserve texture information as much as possible in the captured aircraft skin images, this embodiment of the invention uses cylindrical back-projection to correct the surface curvature of the aircraft skin images, projecting the original image cylindrically onto the tangent plane of the cylinder. Based on the camera parameters, the width and height of the captured image are w and h, the camera's field of view is α, and the focal length is f. The mapping relationship between the captured original image and the image after cylindrical back-projection in the horizontal and vertical directions is as follows: Figure 2 As shown, the geometric relationships are as follows:

[0069]

[0070]

[0071] Where θ represents the angle of the original image pixel, (x ′ ,y ′ () represents the pixel coordinates of the original image. According to formulas (1) and (2), the pixel coordinates (x, y) after inverse cylindrical projection can be obtained:

[0072]

[0073] S12: After cylindrical back projection correction of the image, nearest neighbor interpolation is used to fill in the missing pixels in the image to approximate the distorted areas of the corrected image, and finally obtain the image after distortion correction.

[0074] The pixel value at coordinate (a,b) is: f(a,b)=(f(a-1,b)+f(a+1,b)+f(a,b-1)+f(a,b+1)) / 4.

[0075] Step 202: Using camera position information and image size, calculate the overlapping area between adjacent images to obtain the feature mask;

[0076] Step 202 is as follows:

[0077] S21: During the AGV image acquisition process, the camera moves parallel to the aircraft fuselage. The size of the overlapping area between two images can be determined based on the displacement distance and the dimensions of the acquired images. Then, a mask is used to delete mismatched feature point pairs in the non-overlapping areas, retaining the matching point pairs in the overlapping area. Given that the width and height of the images acquired by the AGV are w and h respectively, the distance from the optical center to the aircraft is z, the horizontal displacement of the AGV between two adjacent images is a, and the width of the overlapping area between adjacent images is P. mask The camera focal length is f, and the physical length of the aircraft skin in the image is x, such as Figure 3 As shown, where l and m represent the physical plane and the imaging plane respectively, the following correspondence can be obtained:

[0078]

[0079] P can be calculated mask .

[0080]

[0081] S22: According to P mask It can obtain the overlapping area between adjacent graphics and set feature masks.

[0082] Step 203: Use the LOFTR algorithm to extract feature points, and then filter the extracted feature points according to the feature mask;

[0083] Specifically, step 203 includes:

[0084] S31: Use the LOFTR algorithm to extract and match feature points in the corrected image, and obtain the confidence level of the matching point pairs;

[0085] S32: Remove feature point pairs from non-overlapping regions based on the feature mask, thereby removing incorrect matching point pairs from non-overlapping regions.

[0086] Step 204: Divide the image into grids according to the idea of ​​image segmentation, and delete redundant feature points in dense feature point areas to make the distribution of feature points more uniform.

[0087] Among them, LoFTR performs well in extracting matching point pairs for weakly textured images. However, this algorithm first selects feature points (corners, edges) with high response values ​​for matching. In aircraft skin images, these areas are relatively few, resulting in fewer detected feature points in weakly textured regions compared to other areas. This leads to uneven spatial distribution of feature points, affecting the accuracy of image registration. Furthermore, the concentrated distribution of feature points generates many redundant feature points, resulting in low stitching efficiency. Based on the idea of ​​image segmentation, according to the confidence of the matching point pairs, only feature points with high confidence are retained in different regions, making the feature point distribution more uniform. Redundant feature points can also be removed, thereby improving the speed and accuracy of image registration. S41: Based on the size of the aircraft skin image, the image can be divided into several 20×20 pixel grids; S42: The confidence (conf) and pixel coordinates of the matching point pairs are obtained through the LoFTR algorithm.

[0088] S43: Based on the confidence level conf, filter all feature points in each grid and retain the three feature points with the highest confidence level;

[0089] S44: Other feature point pairs are treated as redundant feature point pairs and deleted to obtain the final matching point pairs between images.

[0090] Step 205: Use the RANSAC algorithm to select a similarity transformation matrix with the optimal number of interior points to transform the image, making the image registration more accurate.

[0091] In practical implementation, after obtaining accurate feature matching point pairs, it is necessary to calculate the transformation matrix between images. To analyze and inspect the acquired aircraft skin images, it is essential to preserve as much information as possible from multiple viewpoints. Traditional image stitching algorithms typically use homography matrices to perform perspective transformations on the images; however, transforming all images to the same viewpoint results in severe perspective distortion, leading to the loss of image information. To preserve as much of the original information of the aircraft skin as possible, this embodiment of the invention employs similarity transformation to calculate the transformation model, effectively reducing distortion between images and resulting in a more natural image stitching effect.

[0092] Specifically, step 205 includes:

[0093] S51: Use the RANSAC algorithm to remove outliers from the feature points after meshing, and estimate the similarity matrix model based on the best interior points;

[0094] S52: Perform similarity transformation on the image to preserve the aircraft skin texture information as much as possible. Similarity transformation means that the image is only scaled, translated and rotated, so as to maintain the shape and structure of the image.

[0095] The similarity matrix is ​​defined as follows:

[0096]

[0097] The similarity transformation matrix has 4 degrees of freedom, where s is the scaling factor (t... x ,t y ) is the translation vector, and θ is the rotation angle.

[0098] S53: Using the stitched image as the base image, continue stitching other images to achieve multi-image stitching. Step 206: Use a fade-in / fade-out weighted algorithm to eliminate seams between images, achieving seamless stitching.

[0099] Specifically, step 206 includes:

[0100] S61: Calculate the distances d1 and d2 of the pixels within the overlapping region from the left and right ends of the overlapping region based on the transformation matrix;

[0101] S62: Calculate the pixel weights w1 and w2 when fusing the two images based on d1 and d2, where width represents the horizontal coordinate length of the overlapping region. The formulas for calculating w1 and w2 are as follows:

[0102]

[0103] S63: Weighted fusion of pixel values ​​from two images is performed based on w1 and w2, where (x,y) are the image pixel coordinates, I1(x,y) and I2(x,y) are the two images to be fused, and I(x,y) is the pixel value of the overlapping area of ​​the two images after fusion. The calculation formula is as follows:

[0104]

[0105] Ultimately, during the fusion process, w1 gradually changes from 1 to 0, and w2 gradually changes from 0 to 1, allowing the images to transition slowly and smoothly in the overlapping areas, achieving seamless stitching and thus improving the visual effect of image stitching.

[0106] In summary, the embodiments of the present invention solve the problem of poor stitching effect caused by the small number and uneven distribution of feature points detected in aircraft skin images through the above steps 201-206.

[0107] Example 3

[0108] The following are specific examples. Figures 4-7 Tables 1-3 verify the feasibility of the schemes in Examples 1 and 2, as detailed below:

[0109] The experiment tested and verified the images of the aircraft skin acquired by a self-developed unmanned vehicle (AGV). The images used in the experiment were continuously acquired by controlling the movement of the AGV to capture the aircraft skin images. Figure 4 As shown. During the shooting process, the camera was positioned at the same distance from the aircraft, looking directly at it, and five sets of close-up images of the aircraft skin were taken while moving. The camera was about 5 meters away from the fuselage, with seven images in each set, and there were overlapping areas between the images.

[0110] Evaluation metrics description:

[0111] (1) In order to verify that the embodiments of the present invention can extract more accurate matching point pairs, SIFT, ORB, SURF, BRISK and AKAZE and the present method were used respectively to obtain the feature matching results between images;

[0112] (2) When using gridding to filter feature points in an image, the image registration time can be reduced because some redundant feature points are deleted. The average time reduction after adding gridding filtering is statistically analyzed.

[0113] (3) In order to further compare the image registration performance of different algorithms, this method uses structural similarity (SSIM) to measure information such as brightness, contrast and structural similarity of the overlapping area of ​​two images.

[0114] (4) The calculation method for SSIM is as follows:

[0115]

[0116] Where, μ x μ y σ represents the average intensity of the two images, respectively. x σ y Let σ represent the standard deviation of the two images respectively. xy C1 and C2 are the covariances of the two images, where C1 and C2 are constants. Comparison method description: The method provided in this embodiment of the invention is compared with the following five methods:

[0117] (1) SIFT Algorithm: SIFT (scale-invariant feature transform) is a scale-invariant feature extraction method. It is a computer vision algorithm used for feature detection, description, and matching. Invented by David Lowe in 1999, SIFT has become one of the most widely used algorithms in the field of computer vision. The working principle of the SIFT algorithm is to achieve scale invariance by detecting and describing keypoints in an image at different scales. It uses the difference-of-Gaussian pyramid to detect and describe local features and uses an orientation histogram to describe them. SIFT features are rotation, translation, and scale invariant, thus exhibiting high robustness in tasks such as image matching and object recognition. Reference: Lowe and David G. "Distinctive image features from scale-invariant keypoints." International journal of computer vision 60(2004):91-110.

[0118] (2) SURF Algorithm: The SURF (Speeded Up Robust Features) algorithm also possesses scale invariance, rotation invariance, and local invariance. Unlike the SIFT algorithm, the SURF algorithm uses the Hessian matrix to detect key points and uses integral images to accelerate computation. This makes the SURF algorithm perform better when processing large image datasets. Reference: Bay H, Ess A and Tuytelaars T. Speeded-up robust features (SURF)[J]. Computer vision and image understanding, 2008, 110(3):346-359.

[0119] (3) ORB Algorithm: The ORB (Oriented FAST and Rotated BRIEF) algorithm is a computer vision algorithm used to detect and describe local features in images. It was proposed by Ethan Rublee et al. in 2011 and is an algorithm based on the FAST feature detector and the BRIEF feature descriptor. Reference: Rublee, V. Rabaud, K. Konolige and G. Bradski, "ORB: An efficient alternative to SIFT or SURF," 2011 International Conference on Computer Vision, Barcelona, ​​Spain, 2011, pp. 2564-2571.

[0120] (4) BRISK Algorithm: The BRISK (Binary Robust Invariant Scalable Keypoints) algorithm is a fast and robust algorithm for feature point detection and description, suitable for many computer vision tasks. Reference: Leutenegger S, Chli M, Siegwart R Y. BRISK: Binary robust invariant scalable keypoints[C] / / 2011 International conference on computer vision. IEEE, 2011:2548-2555.

[0121] (5) AKAZE Algorithm: The AKAZE (Accelerated-KAZE) algorithm achieves faster speed by reducing the number of layers in the scale space and improving the calculation method of feature descriptors. Reference: Alcantarilla PF, SolutionsT. Fast explicit diffusion for accelerated features in nonlinear scale spaces[J]. IEEE Trans. Patt. Anal.Mach. Intell, 2011, 34(7): 1281-1298.

[0122] Experimental results:

[0123] 1) Use SIFT, ORB, SURF, BRISK, AKAZE, and this method respectively to obtain feature matching results between images, such as... Figure 5 , 6 As shown;

[0124] 2) The image registration time after adding grid filtering is statistically analyzed as shown in Table 1. The stitching speed increased by up to 8.9% and the average speed increased by 4.88%. It can be seen that grid filtering reduces stitching time while ensuring image registration effect.

[0125] After comparing our method with existing methods (SIFT, SURF, ORB, BRISK, and AKAZE), the matching rates of SIFT, SURF, ORB, BRISK, and AKAZE were 4.84%, 0.47%, 2.9%, 0.86%, and 5.08%, respectively. The proposed algorithm achieved a feature matching rate of 55.21%. Statistical results are shown in Table 2. The average SSIM score of our method was 82.83%, while the average SSIM scores of SIFT, SURF, ORB, BRISK, and AKAZE were 48.98%, 43.95%, 57.37%, 54.89%, and 47.00%, respectively. Our method improved the average SSIM score by 44.38%–88.46% compared to existing methods (SIFT, SURF, ORB, BRISK, and AKAZE). Statistical results are shown in Table 3. Results for some images are shown below. Figure 7 As shown, when using the above five feature extraction algorithms to extract feature points from aircraft skin, the aircraft skin image contains many weak texture areas, resulting in a small number of detected feature point pairs, leading to poor image registration and serious missing stitching problems, making it impossible to complete the image stitching task. In contrast, the method described above can obtain a large number of feature points in this scenario, thereby making the image registration more accurate and avoiding the problems of missing or incorrect stitching. Figure 7 (f) demonstrates the effect of LoFTR with a feature mask on stitching together acquired aircraft skin images. The black background at the bottom right of the image is 17 pixels wide. This illustrates the effect of this method on stitching together acquired aircraft skin images, compared to... Figure 7 (f) The black background at the bottom right of the image is only 5 pixels wide, which reduces the distortion caused by splicing.

[0126] Table 1

[0127]

[0128] Table 2

[0129]

[0130] Table 3

[0131]

[0132] Example 4

[0133] An image stitching device for weakly textured curved surfaces facing aircraft skin, the device comprising: a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the following method:

[0134] Based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original image, and nearest neighbor interpolation is used to fill in the missing pixels in the image.

[0135] Using camera position information and image size, the overlapping area between adjacent images is calculated to obtain a feature mask;

[0136] The LOFTR algorithm is used to extract feature points, and feature point pairs in non-overlapping regions are removed based on the feature mask.

[0137] The image is divided into grids based on the idea of ​​image segmentation, and redundant feature points in dense feature point areas are deleted;

[0138] The RANSAC algorithm is used to select a similarity transformation matrix with the optimal number of interior points to transform the image, and fade-in / fade-out weighted methods are used to eliminate seams between images, achieving seamless stitching.

[0139] Specifically, cylindrical back projection is used to correct the acquired original image, and nearest neighbor interpolation is used to complete the missing pixels in the image.

[0140] Based on the camera parameters, the width and height of the acquired image are w and h, respectively, the camera's field of view is α, and the focal length is f. The mapping relationship between the acquired original image and the image after cylindrical back projection in the horizontal and vertical directions is as follows:

[0141]

[0142]

[0143] Where θ represents the skew angle of the original image pixel, and (x′, y′) represents the pixel coordinates of the original image. According to formulas (1) and (2), the pixel coordinates (x, y) after inverse cylindrical projection can be obtained:

[0144]

[0145] Nearest neighbor interpolation is used to fill in the missing pixels in the image and approximate the distorted areas of the corrected image, thus obtaining the image after distortion correction.

[0146] The pixel value at coordinate (a,b) is: f(a,b)=(f(a-1,b)+f(a+1,b)+f(a,b-1)+f(a,b+1)) / 4.

[0147] Specifically, by using camera position information and image size, the overlapping region between adjacent images is calculated, resulting in the feature mask:

[0148] The size of the overlapping area between the two images is determined based on the displacement distance and the size of the acquired images. By setting a mask, mismatched feature point pairs in the non-overlapping area are deleted, while matching point pairs in the overlapping area are retained.

[0149] Given that the width and height of the image acquired by the AGV are w and h respectively, the distance from the optical center to the aircraft is z, the horizontal displacement of the AGV between two adjacent images is a, and the width of the overlapping area between adjacent images is P. mask The camera focal length is f, and the physical length of the aircraft skin in the image is x, where l and m represent the physical plane and the imaging plane, respectively. The following correspondence is obtained.

[0150]

[0151]

[0152] According to P mask Obtain the overlapping area between adjacent graphics and set a feature mask.

[0153] Furthermore, based on the idea of ​​image segmentation, the image is divided into grids, and redundant feature points in densely populated regions are removed, specifically as follows:

[0154] Based on the size of the aircraft skin image, the image is divided into several 20×20 pixel grids;

[0155] The confidence score (conf) and pixel coordinates of the matching point pairs are obtained using the LoFTR algorithm.

[0156] Based on the confidence level (conf), all feature points within each grid are filtered, and the three feature points with the highest confidence levels are retained.

[0157] Other feature point pairs are treated as redundant and deleted to obtain the final matching point pairs between images.

[0158] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0159] The execution entities of the processor 1 and memory 2 mentioned above can be devices with computing functions such as computers, microcontrollers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution entity and can select it according to the needs of actual application.

[0160] Data signals are transmitted between memory 2 and processor 1 via bus 3, which will not be described in detail in this embodiment of the invention.

[0161] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0162] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0163] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0164] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0165] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0166] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for stitching weakly textured surface images for aircraft skin, characterized in that, The method includes: Based on the position information of the camera and the aircraft model, cylindrical back projection is used to correct the acquired original image, and nearest neighbor interpolation is used to fill in the missing pixels in the image. Using camera position information and image size, the overlapping area between adjacent images is calculated to obtain a feature mask; Feature points are extracted using the LOFTR algorithm, and feature point pairs in non-overlapping regions are removed based on the feature mask. The image is divided into grids based on the idea of ​​image segmentation, and redundant feature points in dense feature point areas are deleted; The RANSAC algorithm is used to select a similarity transformation matrix with the optimal number of interior points to transform the image, and fade-in and fade-out weighted methods are used to eliminate seams between images to achieve seamless stitching. The feature mask is obtained by calculating the overlapping area between adjacent images using camera position information and image size: The size of the overlapping area between the two images is determined based on the displacement distance and the size of the acquired images. By setting a mask, mismatched feature point pairs in the non-overlapping area are deleted, while matching point pairs in the overlapping area are retained. Given that the width and height of the image acquired by the AGV are respectively The distance from the optical center to the airplane is The horizontal displacement of the AGV between two adjacent images is The image width of the overlapping region between adjacent images is The camera focal length is The physical length of the aircraft skin in the image is ,in Representing the physical plane and the imaging plane respectively, the following correspondence is obtained: ； ； according to Obtain the overlapping area between adjacent graphics and set a feature mask.

2. The method for stitching weakly textured surface images for aircraft skin according to claim 1, wherein... special The feature is that the method of correcting the acquired original image using cylindrical back projection and using nearest neighbor interpolation to fill in the missing pixels in the image specifically involves: Based on the camera's parameters, the width and height of the acquired image are respectively... The camera's field of view is focal length is The mapping relationship between the acquired original image and the image after cylindrical backprojection in the horizontal and vertical directions is as follows: ； （1） ；（2） in, This represents the angle of a pixel in the original image. The pixel coordinates of the original image are represented by formulas (1) and (2), and the pixel coordinates after inverse cylindrical projection are obtained. ): ； Nearest neighbor interpolation is used to fill in the missing pixels in the image and approximate the distorted areas of the corrected image, thus obtaining the image after distortion correction. Where the coordinates are The pixel value is: 。 3. The method for stitching weakly textured surface images for aircraft skin according to claim 1, characterized in that, The process of dividing the image into grids based on the idea of ​​image segmentation and deleting redundant feature points in densely populated regions specifically involves: Based on the size of the aircraft skin image, the image is divided into several 20×20 pixel grids; The confidence level of the matching point pairs is obtained using the LoFTR algorithm. and pixel coordinates; Based on confidence level For each grid cell, all feature points are filtered, and the three feature points with the highest confidence are retained. Other feature point pairs are treated as redundant and deleted to obtain the final matching point pairs between images.

4. A device for stitching images of weakly textured curved surfaces facing aircraft skin, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-3.

Images