Method for fast stitching and acquisition of large-size structure image data of aerospace

Abstract

This invention discloses a rapid image stitching acquisition method for large-size aerospace structures, comprising: acquiring original images of the workpiece and performing vertical stitching to obtain a sequence of vertically stitched images; dividing the overlapping areas of adjacent vertically stitched images into blocks, using a deep learning network to extract and match features from image block pairs, determining local matching point pairs and restoring coordinates to form a global matching point pair set; calculating the affine transformation matrix of adjacent images based on the global matching point pair set, and then obtaining the cumulative affine transformation matrix using the first image as a reference; projecting each image onto a global canvas and fusing them at once according to the cumulative affine transformation matrix to obtain a complete workpiece surface image; finally, performing sliding window detection to identify small-size targets on the vertically stitched images, and performing whole-image detection to identify large-size targets on the complete image, and statistically storing the target information. This invention overcomes the problems of feature loss in high aspect ratio image stitching, accumulation of multi-image stitching errors, and missed detection of multi-scale targets.

Description

Technical Field

[0001] This invention belongs to the field of large workpiece splicing recognition in the aerospace industry, and particularly relates to a method for rapid splicing and acquisition of large-size aerospace structural image data. Background Technology

[0002] In the automated visual inspection of large-sized aerospace components (such as cabin hulls and box bottoms), the dimensions of the components far exceed the camera's field of view, necessitating the reconstruction of the complete surface through multi-image stitching. Therefore, existing technologies typically suffer from the following drawbacks: Image feature loss due to large aspect ratio leads to stitching failure: Images captured by line scan cameras and vertically stacked have extremely large aspect ratios (the height can be tens of times the width). When using deep learning models for feature matching, existing technologies usually need to scale the entire image to the fixed input size of the model. This global scaling will cause the texture in the vertical direction to be severely compressed and blurred, and the feature point extraction will be inaccurate, which will lead to registration failure or misalignment between adjacent images.

[0003] Error accumulation caused by sequential stitching of multiple images: Traditional multi-image stitching usually adopts a "chain" process, that is, first stitching image A with image B to obtain the result image AB, and then stitching the result image AB with image C. Each stitching involves image resampling (warping) and interpolation. As the number of images increases, the number of resampling times increases linearly, causing image details to gradually become blurred. Moreover, the registration error in the previous step will be propagated and amplified to subsequent steps, ultimately reducing the geometric accuracy of the large image.

[0004] A single detection model cannot handle multi-scale targets simultaneously: the workpiece surface contains both small targets (such as rivets) and large targets (such as windows and support strips). If the entire image after stitching is directly inspected, small targets are easily missed due to their small size; if only slice detection is used, large targets may be cut and destroyed.

[0005] This shows that existing technologies lack adaptive detection and coordinate unification mechanisms for targets of different scales. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a rapid stitching and acquisition method for large-size aerospace structural image data, aiming to achieve high-precision, low-error stitching and reconstruction of large-size images and multi-scale target recognition.

[0007] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution: A method for rapid stitching and acquisition of large-size structural image data in aerospace, characterized by the following steps: Original images of the workpiece are acquired from different poses. At least two original images are acquired from each pose and each original image is numbered. The original images acquired from the same pose are sorted in ascending order using the number as the sorting key to obtain the sequence of original images of the workpiece in each pose. For each workpiece original image sequence, the original images of each workpiece are scaled based on a preset scaling factor and maximum width constraint, and horizontally mirrored according to the flip mark. The scaled and flipped workpiece images are then placed vertically onto a blank canvas in the order of arrangement to obtain a vertically stitched image. All workpiece original image sequences are traversed to obtain a vertically stitched image sequence. For two adjacent vertically stitched images, calculate the common height of the two vertically stitched images in the vertical direction. Based on the preset block height and the overlapping pixels between blocks, divide the common area into image block pairs with overlapping areas along the vertical direction. Perform feature extraction and matching on each image block pair to obtain local matching point pairs. Restore the coordinates of each local matching point pair to the original image coordinate system. Merge all local matching point pairs after coordinate restoration to form a global matching point pair set of adjacent vertically stitched images. Based on the global matching point pair set, calculate the affine transformation matrix between adjacent vertically stitched images; Using the first vertically stitched image in the sequence as the reference image, the cumulative affine transformation matrix of the other vertically stitched images relative to the reference image is calculated. Based on the cumulative affine transformation matrix, the corner points of the other vertically stitched images are transformed to the global coordinate system, and the global canvas size and offset are determined. The global coordinate system is the coordinate system of the reference image. Based on the offset and the cumulative affine transformation matrix, a perspective transformation matrix is ​​constructed to transform the other vertically stitched images. Figure 1 The image is projected onto the global canvas in one step, and the overlapping areas are then fused using a weighted average to obtain a stitched image of the complete workpiece surface. Small-sized targets on the workpiece surface are detected based on vertically stitched image sequences, and large-sized targets on the workpiece surface are obtained based on complete workpiece surface images. The categories, locations, and quantities of different-sized detected targets are statistically analyzed and saved.

[0008] Furthermore, the scaling of the original images of each workpiece based on a preset scaling factor and a maximum width constraint specifically involves: Preset scaling factor and maximum width First, scale the original image according to the scaling factor. Scaling is performed; if the width of the scaled image exceeds... Then The scaling factor is the ratio of the original image width to the actual scaling ratio; if it does not exceed this ratio, the image is scaled according to the scaling factor. The scaled image.

[0009] Furthermore, the calculation of the common height of the two vertically stitched images in the vertical direction, and the division of the common area into image block pairs with overlapping areas along the vertical direction based on the preset block height and the overlapping pixels between blocks, specifically involves: Let the two adjacent vertically stitched images be the left image. Right Figure ;Will , Convert a color image to a grayscale image and calculate its common height. Based on the preset block height B and the overlapping pixels O between blocks, image blocks are generated within a common height range, i.e., the block segmentation process satisfies... and , This represents the starting height of any image block; if the remaining height is insufficient to create another image block of height B, then the remaining height is used for block creation.

[0010] Furthermore, the step of extracting and matching features for each image patch pair to obtain local matching point pairs specifically involves: Set an overlap ratio Left image Preserve its width direction from arrive The area is used as the matching area in the left image, and the right image... Preserve its width direction from 0 to The region is designated as the matching area in the right image, and feature extraction and matching are performed only within the matching areas in the left and right images. , These are the widths of the left and right images, respectively. For the image information in the matching areas of the left and right images, the images are scaled according to the input size of the SuperPoint network in the form of a pair of image blocks, and then input into the SuperPoint network to obtain the feature point coordinates and descriptor matrices of the left and right image blocks. The feature point coordinates and descriptor matrices are then input into the SuperGlue network for feature matching to obtain local matching point pairs between the left and right image blocks.

[0011] Furthermore, the step of using the first vertically stitched image in the vertically stitched image sequence as the reference image to calculate the cumulative affine transformation matrix of the other vertically stitched images relative to the reference image specifically involves: For vertical stitching sequence The affine transformation matrix between adjacent vertically stitched images ,in This indicates the vertical stitching pattern starting from the i-th image. The affine transformation matrix for the (i-1)th vertically stitched image is first... Expanded into a homogeneous matrix Then, perform the cumulative transformation according to the following formula: ; in, This indicates the vertical stitching pattern starting from the i-th image. To the baseline map The cumulative affine transformation matrix, Then it is the identity matrix.

[0012] Furthermore, the step of transforming the corner points of other vertically stitched images to the global coordinate system based on the cumulative affine transformation matrix, and determining the global canvas size and offset, specifically involves: For the i-th vertical stitching image Through the cumulative affine transformation matrix Transform its four corner points to the global coordinate system; traverse all vertically stitched images to obtain the minimum x-coordinate of the corner points in the global coordinate system. Minimum y-coordinate Maximum x-coordinate Maximum ordinate Then, expand outwards by a preset extension pixel d to obtain the boundary coordinates of the global canvas. The global canvas size, obtained based on boundary coordinates, is: canvas width. Canvas height Then, based on the bottom left boundary point of the global canvas, confirm the horizontal axis offset. Vertical axis offset .

[0013] Furthermore, the perspective transformation matrix is ​​constructed based on the offset and the cumulative affine transformation matrix, and other vertical stitching is then performed. Figure 1 The image is projected onto the global canvas in one step, and the overlapping areas are then weighted and averaged to obtain the stitched complete workpiece surface image. floating-point canvas As a global canvas, the fusion weight matrix weight is initialized to 0; Each vertically stitched image is projected onto the global canvas through a perspective transformation. The perspective transformation matrix is ​​a cumulative affine transformation matrix with an offset, expressed by the formula: ; in, The i-th vertical stitching image To perspective transformation projection diagram The perspective transformation matrix; For each perspective transformation projection image Generate a binary mask: non-zero pixel regions are 1, and zero pixel regions are 0; use the binary mask as... Fusion weights ; The pixel values ​​of each perspective transformation projection image are accumulated into the floating-point canvas, and the fusion weights of each perspective transformation projection image are accumulated into the weight matrix. The pixel values ​​of each point in the accumulated floating-point canvas are divided by the accumulated weights of each point in the weight matrix to obtain the complete workpiece surface image after weighted average fusion.

[0014] Furthermore, the specific steps of detecting small-sized targets on the workpiece surface based on vertically stitched image sequences, acquiring large-sized targets on the workpiece surface based on complete workpiece surface images, and statistically storing the categories, locations, and quantities of detected targets of different sizes are as follows: For each vertically stitched image , generate A square sliding window with a width equal to the side length slides vertically at a preset overlap rate; target detection inference is performed on the image information within each sliding window, and target detection boxes are marked. Restore the target detection bounding box to the vertical mosaic coordinate system. Then, the target detection boxes are mapped to the global coordinate system through the cumulative affine transformation matrix; all mapped target detection boxes are collected, and non-maximum suppression is performed in batches in the global coordinate system to obtain the detection results of small-scale targets; Full-image target detection inference is performed on the complete workpiece surface image to obtain the detection results of large-scale targets; The detection results of targets at different scales are combined and plotted on the complete workpiece surface image, and the number and location of each type of target are counted.

[0015] Based on the above technical solution, the present invention has at least the following beneficial effects: By employing an automatic block segmentation strategy, images with large aspect ratios are divided into multiple overlapping blocks for feature extraction and matching, avoiding texture blurring caused by global scaling and significantly improving the robustness of feature matching and stitching accuracy under extreme aspect ratio images.

[0016] By accumulating transformation matrices and performing a one-time fusion, the cumulative transformation matrix of all images relative to the reference image is first calculated. Finally, only the original image is subjected to a single perspective transformation and projected onto the global canvas. This eliminates the resampling step in the intermediate process, avoids error accumulation, and ensures the geometric fidelity of the final large image.

[0017] By combining sliding window detection with whole-image detection and unified coordinate mapping, sliding window concurrent inference is used for small targets, while whole-image inference is used for large targets. A cumulative transformation matrix is ​​then used to uniformly map the detection results of each sub-image to the global coordinate system. This achieves high-precision simultaneous identification of minute rivets and large structural components within the same detection process, without interference from splicing traces. Attached Figure Description

[0018] Figure 1This is an overall flowchart of the rapid stitching and acquisition method for large-size aerospace structural image data proposed in this invention. Figure 2 This is a diagram showing the vertical splicing result in the method proposed in this invention; Figure 3 This is a schematic diagram of the block processing process in the method proposed in this invention; Figure 4 This is a schematic diagram of the feature matching process in the method proposed in this invention; Figure 5 This is a diagram showing the horizontal stitching result of the method proposed in this invention; Figure 6 This is the result of multi-image cumulative stitching in the method proposed in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0021] This embodiment proposes a rapid stitching and acquisition method for large-size aerospace structural image data, which can be deployed in a large-size aerospace workpiece visual inspection system. The workpiece visual inspection system includes a control console, multi-stage lifting columns, a robotic arm, a rotary table, and a truss system. A line scan camera is mounted at the end of the robotic arm to acquire raw images of the workpiece surface from multiple poses. Figure 1 As shown, the method proposed in this invention specifically includes the following steps: Original images of the workpiece are acquired from different poses. At least two original images are acquired from each pose and each original image is numbered. The original images acquired from the same pose are sorted in ascending order using the number as the sorting key to obtain the sequence of original images of the workpiece in each pose. In this embodiment, 12 sets of original workpiece images (i.e., 12 sequences of original workpiece images) are acquired by a line scan camera. Each set contains 24 original images, including 23 full images with a resolution of 4096*5000 and 1 half image with a width of 4096 and a height of less than 5000.

[0022] For each workpiece's original image sequence, the original images of each workpiece are scaled based on a preset scaling factor and a maximum width constraint, and then horizontally mirrored according to the indication of the flip mark. In a preferred embodiment, the scaling of the original images of each workpiece based on a preset scaling factor and a maximum width constraint specifically involves: Preset scaling factor ( ) and maximum width First, scale the original image according to the scaling factor. Scaling is performed; if the width of the scaled image exceeds... Then The scaling factor is the ratio of the original image width to the actual scaling ratio; if it does not exceed this ratio, the image is scaled according to the scaling factor. The scaled image is used to maximize the preservation of image details while ensuring processing feasibility, and to ensure that the output image size is controllable and that the horizontal stitching effect is not severely affected by image blur. Regarding horizontal mirror flipping, in actual operation, the robotic arm holds the line scan camera and aligns it above the large-sized aerospace workpiece on the rotary table. The rotary table rotates 360 degrees clockwise to obtain an original image covering 0 to 360 degrees horizontally. Then, the robotic arm holds the line scan camera and moves vertically downwards, while the rotary table rotates 360 degrees counterclockwise to obtain another original image covering 360 to 0 degrees horizontally. The robotic arm continues to move vertically, rotating the rotary table clockwise, and so on. The resulting original image has two orientations. Due to the clockwise and counterclockwise rotation of the rotary table, the original image obtained by counterclockwise rotation, after horizontal mirror flipping, will have the same horizontal orientation as the image obtained by clockwise rotation. Therefore, in this embodiment, the following settings can be made: the flip flag for the original image obtained by clockwise rotation of the rotary table is set to true, and the flip flag for the original image obtained by counterclockwise rotation is set to false. Horizontal mirror flipping is performed when the flip flag corresponding to the original image is false; this horizontal mirror flipping maintains the visual consistency of the image.

[0023] The scaled and flipped workpiece images are sequentially placed vertically onto a blank canvas in their arrangement order to obtain a vertically stitched image. This process is repeated for all original workpiece image sequences to obtain a sequence of vertically stitched images. In this embodiment, vertical stitching is performed on 12 sets of original workpiece images, with the maximum image width set to 512. The stitching results are then saved sequentially in their respective subfolders. The vertically stitched images are shown below. Figure 2 As shown.

[0024] In image stitching tasks, image registration is a core step. Due to the variable size of the workpiece and the limited field of view of the camera, multiple sets of images are captured, and then stitched layer by layer based on the common parts between adjacent images. However, the aspect ratio of the stitched image obtained after vertical stitching is usually tens of times. Direct horizontal stitching requires scaling the image before feature extraction and matching. However, the size difference between the horizontal and vertical directions after scaling will cause severe image blurring, affecting the horizontal stitching effect. Therefore, this invention considers using an automatic block segmentation strategy and cumulative affine transformation to achieve highly robust image registration and stitching.

[0025] For two adjacent vertically stitched images, calculate the common height of the two vertically stitched images in the vertical direction. Based on the preset block height and the overlapping pixels between blocks, divide the common area into image block pairs with overlapping areas along the vertical direction. In a preferred embodiment, the step of calculating the common height of the two vertically stitched images in the vertical direction, and dividing the common area into image block pairs with overlapping areas along the vertical direction based on the preset block height and the overlapping pixels between blocks, specifically involves: Let the two adjacent vertically stitched images be the left image. Right Figure ;Will , Convert a color image to a grayscale image and calculate its common height. Based on the preset block height B and the overlapping pixels O between blocks, image blocks are generated within a common height range, i.e., the block segmentation process satisfies... and , This represents the starting height of any image block; if the remaining height is insufficient to create another image block of height B, then the remaining height is used for block creation. For example... Figure 3 As shown, after segmentation, multiple image blocks are obtained. The height of each image block is the same as the size of the original image. Only horizontal cropping is performed. A common area of ​​0.2 times the width of the image block is set between adjacent image blocks (shown in the blue frame). Subsequently, the algorithm is used to extract and match features from the image blocks.

[0026] In this embodiment, the height of the image blocks can be dynamically adjusted based on the aspect ratio of the vertically stitched image and the accuracy requirements of the object to be recognized. For example, to achieve a better stitching effect, each block can be square, with the width of the input image used as the block height. For applications with a particularly large aspect ratio or lower accuracy requirements for the object to be recognized, the block height can be appropriately increased; for example, the block height can be a multiple of the input image width. times, of which the common width is .

[0027] Feature extraction and matching are performed on each image patch pair to obtain local matching point pairs; In a preferred embodiment, the step of extracting and matching features from each image block pair to obtain local matching point pairs specifically involves:

[0028] Set an overlap ratio In this embodiment, we take Left image Preserve its width direction from arrive The area is used as the matching area in the left image, and the right image... Preserve its width direction from 0 to The region is designated as the matching area in the right image, and feature extraction and matching are performed only within the matching areas in the left and right images. , The widths are set for the left and right images, respectively; this ensures that subsequent left and right image blocks cover the main texture of the overlapping area, eliminating interference from similar area features of the workpiece.

[0029] For the image information in the matching areas of the left and right images, the images are scaled according to the input size of the SuperPoint network as a pair of image blocks, and then input into the SuperPoint network to obtain the feature point coordinates and descriptor matrices of the left and right image blocks. These feature point coordinates and descriptor matrices are then input into the SuperGlue network for feature matching to obtain local matching point pairs between the left and right image blocks, such as... Figure 4 As shown in the figure, the green lines represent interior points. These matching point pairs have passed the RANSAC (Random Sample Consensus) algorithm and are considered to conform to the overall image transformation model (affine matrix). Therefore, they are used to calculate the final transformation matrix (such as findHomography or estimateAffine2D) and are reliable local matching point pairs. The red lines represent exterior points. These matching point pairs are judged by RANSAC as incorrect matches that do not conform to the transformation model (e.g., due to repeated textures, occlusion, or inconsistent motion). They do not participate in the calculation of the final transformation matrix and are discarded during stitching. Restore the coordinates of each local matching point pair to the original image coordinate system; merge all the local matching point pairs after coordinate restoration to form a global matching point pair set of adjacent vertical stitched images, denoted as { }、{ }, , This is a pair of globally matching points.

[0030] Based on the globally matched point pair set, the affine transformation matrix between adjacent vertically stitched images is calculated. In this embodiment, the `estimateAffine2D` function of OpenCV is used to obtain the 2×3 affine transformation matrix A from the right image to the left image. This matrix satisfies: At this point, based on the affine transformation matrix, horizontal splicing can actually be performed to obtain, as shown below. Figure 5 The image shown is a horizontally stitched image. However, in the traditional chain-style horizontal stitching method, each stitch causes the image details to gradually blur, and the registration error of the previous step will be transmitted and amplified to the subsequent steps, ultimately reducing the geometric accuracy of the large image. Therefore, after obtaining the affine transformation matrix, this application does not directly perform horizontal stitching sequentially, but selects all images for one-time multi-image cumulative stitching. At the same time, in actual operation, after obtaining the affine transformation, the save path of each vertical stitched image should be retained to facilitate subsequent processing. Multi-image cumulative stitching is based on the transformation matrix between pre-saved adjacent images. By calculating the cumulative transformation matrix, multiple images are merged into a large image at once. The advantage of this calculation method is that it directly accumulates the transformation matrices of adjacent images obtained by feature extraction and matching of the original images, avoiding the introduction of other stitching errors.

[0031] The process of accumulating and stitching multiple images specifically includes: Using the first vertically stitched image in the vertical stitched image sequence as the reference image, calculate the cumulative affine transformation matrix of the other vertically stitched images relative to the reference image; In a preferred embodiment, the step of using the first vertically stitched image in the vertically stitched image sequence as the reference image and calculating the cumulative affine transformation matrix of the other vertically stitched images relative to the reference image specifically involves: For vertical stitching sequence The affine transformation matrix between adjacent vertically stitched images ,in This indicates the vertical stitching pattern starting from the i-th image. The affine transformation matrix up to the (i-1)th vertically stitched image is used to facilitate subsequent multiplication operations (using matrix multiplication) for cumulative stitching of multiple images. First, the affine transformation matrix is... Expanded into a homogeneous matrix Then, perform the cumulative transformation according to the following formula: ; in, This indicates the vertical stitching pattern starting from the i-th image. To the baseline map The cumulative affine transformation matrix, Then it is the identity matrix.

[0032] Based on the cumulative affine transformation matrix, the corner points of other vertically stitched images are transformed to the global coordinate system to determine the global canvas size and offset; the global coordinate system is the coordinate system of the reference image. In a preferred embodiment, the step of transforming the corner points of other vertically stitched images to the global coordinate system based on the cumulative affine transformation matrix, and determining the global canvas size and offset, specifically involves: For the i-th vertical stitching image Through the cumulative affine transformation matrix Transform its four corner points to the global coordinate system; traverse all vertically stitched images to obtain the minimum x-coordinate of the corner points in the global coordinate system. Minimum y-coordinate Maximum x-coordinate Maximum ordinate Then, expand outwards by a preset extension pixel d to obtain the boundary coordinates of the global canvas. The global canvas size, obtained based on boundary coordinates, is: canvas width. Canvas height Then, based on the bottom left boundary point of the global canvas, confirm the horizontal axis offset. Vertical axis offset .

[0033] Construct a perspective transformation matrix based on the offset and cumulative affine transformation matrix, and then stitch other vertical elements together. Figure 1 The image is projected onto the global canvas in one step, and the overlapping areas are then fused using a weighted average to obtain a stitched image of the complete workpiece surface. In a preferred embodiment, this step specifically involves: The perspective transformation matrix is ​​constructed based on the offset and the cumulative affine transformation matrix, and other vertical stitching is then performed. Figure 1 The image is projected onto the global canvas in one step, and the overlapping areas are then weighted and averaged to obtain the stitched complete workpiece surface image. floating-point canvas As a global canvas, the fusion weight matrix weight is initialized to 0; Each vertically stitched image is projected onto the global canvas through a perspective transformation. The perspective transformation matrix is ​​a cumulative affine transformation matrix with an offset, expressed by the formula: ; in, The i-th vertical stitching image To perspective transformation projection diagram The perspective transformation matrix; For each perspective transformation projection image Generate a binary mask: non-zero pixel regions are 1, and zero pixel regions are 0; use the binary mask as... Fusion weights ; The pixel values ​​of each perspective transformation projection are accumulated into the floating-point canvas, and the fusion weights of each perspective transformation projection are accumulated into the weight matrix. The pixel value of each point in the accumulated floating-point canvas is divided by its corresponding accumulated weight in the weight matrix to obtain the weighted average fused complete workpiece surface image. This achieves average fusion of overlapping areas and direct coverage of non-overlapping areas. The final fusion result is as follows: Figure 6 As shown.

[0034] After obtaining the complete workpiece surface image, target recognition can be performed. In this application, small-sized targets (such as rivets) on the workpiece surface are detected based on the vertical stitching image sequence, and large-sized targets (such as holes / stripes) on the workpiece surface are obtained based on the complete workpiece surface image. The categories, locations and quantities of different-sized detected targets are counted and saved. In a preferred embodiment, the target detection process specifically includes: For each vertically stitched image , generate A square sliding window with a width equal to the side length slides vertically at a preset overlap rate; target detection inference is performed on the image information within each sliding window to mark the target detection box; in this embodiment, multi-threaded parallel inference is performed using the YOLOv8 model to obtain a small-sized target detection box. Restore the target detection bounding box to the vertical mosaic coordinate system. Then, the target detection boxes are mapped to the global coordinate system through the cumulative affine transformation matrix; all mapped target detection boxes are collected, and non-maximum suppression is performed in batches in the global coordinate system to obtain the detection results of small-scale targets; Full-image target detection inference is performed on the complete workpiece surface image (using the YOLOv8 model) to obtain the detection results of large-scale targets; The detection results of targets at different scales are combined and plotted on the complete workpiece surface image, and the number and location of each type of target are counted.

[0035] Since the rivets are small and densely distributed, direct inference from the whole image can easily lead to missed detection of small targets. Therefore, a sliding window strategy is adopted. However, the holes / stripes are larger and have regular shapes, so detection inference can be performed directly on the complete workpiece surface image obtained by accumulating and stitching multiple images.

[0036] In summary, this invention proposes a rapid stitching and acquisition method for large-size aerospace structural image data, which solves the problems in existing technologies such as stitching failure caused by loss of image features in large aspect ratio images, error accumulation caused by sequential stitching of multiple images, and the inability of a single detection model to take into account multi-scale targets. It achieves high-precision, low-error stitching and restoration of large-size images and multi-scale target recognition.

[0037] In this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to at least one embodiment or example described in connection with a specific feature, structure, material, or characteristic. These specific features, structures, materials, or characteristics may be combined in a suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples and their features described in this specification.

[0038] The logic and / or steps shown in the flowchart or otherwise described can be viewed as a sequence of executable instructions for implementing logical functions. These instructions may be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. Such systems, apparatus, or devices include processor systems or other systems capable of receiving and executing instructions.

[0039] The above embodiments detail the principles and implementation methods of the present invention, and illustrate its working principle using specific examples. These examples are only used to help understand the method and core ideas of the present invention. Furthermore, based on the ideas of the present invention, actual implementation methods and application scope may vary. Therefore, the content of this specification should not be construed as limiting the present invention.

Claims

1. A method for rapid stitching and acquisition of large-size structural image data in aerospace, characterized in that, Specifically, the following steps are included: Original images of the workpiece are acquired from different poses. At least two original images are acquired from each pose and each original image is numbered. The original images acquired from the same pose are sorted in ascending order using the number as the sorting key to obtain the sequence of original images of the workpiece in each pose. For each workpiece original image sequence, the original images of each workpiece are scaled based on a preset scaling factor and maximum width constraint, and horizontally mirrored according to the flip mark. The scaled and flipped workpiece images are then placed vertically onto a blank canvas in the order of arrangement to obtain a vertically stitched image. All workpiece original image sequences are traversed to obtain a vertically stitched image sequence. For two adjacent vertically stitched images, calculate the common height of the two vertically stitched images in the vertical direction. Based on the preset block height and the overlapping pixels between blocks, divide the common area into image block pairs with overlapping areas along the vertical direction. Perform feature extraction and matching on each image block pair to obtain local matching point pairs. Restore the coordinates of each local matching point pair to the original image coordinate system. Merge all local matching point pairs after coordinate restoration to form a global matching point pair set of adjacent vertically stitched images. Based on the global matching point pair set, calculate the affine transformation matrix between adjacent vertically stitched images; Using the first vertically stitched image in the sequence as the reference image, the cumulative affine transformation matrix of the other vertically stitched images relative to the reference image is calculated. Based on the cumulative affine transformation matrix, the corner points of the other vertically stitched images are transformed to the global coordinate system to determine the global canvas size and offset. The global coordinate system is the coordinate system where the reference image is located. Based on the offset and the cumulative affine transformation matrix, a perspective transformation matrix is ​​constructed to project the other vertically stitched images onto the global canvas at once. The overlapping areas are then weighted and averaged to obtain the complete workpiece surface image after stitching. Small-sized targets on the workpiece surface are detected based on vertically stitched image sequences, and large-sized targets on the workpiece surface are obtained based on complete workpiece surface images. The categories, locations, and quantities of different-sized detected targets are statistically analyzed and saved.

2. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 1, characterized in that, The scaling of the original images of each workpiece based on a preset scaling factor and a maximum width constraint is specifically as follows: Preset scaling factor and maximum width ; First, adjust the original image according to the scaling factor. Scaling is performed; if the width of the scaled image exceeds... Then The scaling factor is the ratio of the original image width to the actual scaling factor. If it does not exceed the limit, it will be retained according to the scaling factor. The scaled image.

3. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 1, characterized in that, The calculation of the common height of the two vertically stitched images in the vertical direction, and the division of the common area into image block pairs with overlapping areas along the vertical direction based on the preset block height and the overlapping pixels between blocks, specifically involves: Let the two adjacent vertically stitched images be the left image. Right Figure ;Will , Convert a color image to a grayscale image and calculate its common height. Based on the preset block height B and the overlapping pixels O between blocks, image blocks are generated within a common height range, i.e., the block segmentation process satisfies... and , This represents the starting height of any image block; if the remaining height is insufficient to create another image block of height B, then the remaining height is used for block creation.

4. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 3, characterized in that, The specific steps of extracting and matching features for each image patch pair to obtain local matching point pairs are as follows: Set an overlap ratio Left image Preserve its width direction from arrive The area is used as the matching area in the left image, and the right image... Preserve its width direction from 0 to The region is designated as the matching area in the right image, and feature extraction and matching are performed only within the matching areas in the left and right images. , These are the widths of the left and right images, respectively. For the image information in the matching areas of the left and right images, the images are scaled according to the input size of the SuperPoint network in the form of a pair of image blocks, and then input into the SuperPoint network to obtain the feature point coordinates and descriptor matrices of the left and right image blocks. The feature point coordinates and descriptor matrices are then input into the SuperGlue network for feature matching to obtain local matching point pairs between the left and right image blocks.

5. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 1, characterized in that, The specific steps for calculating the cumulative affine transformation matrix of other vertically stitched images relative to the reference image, using the first vertically stitched image in the sequence as the reference image, are as follows: For vertical stitching sequence The affine transformation matrix between adjacent vertically stitched images ,in This indicates the vertical stitching pattern starting from the i-th image. The affine transformation matrix for the (i-1)th vertically stitched image is first... Expanded into a homogeneous matrix Then, perform the cumulative transformation according to the following formula: ； in, This indicates the vertical stitching pattern starting from the i-th image. To the baseline map The cumulative affine transformation matrix, Then it is the identity matrix.

6. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 5, characterized in that, The step of transforming the corner points of other vertically stitched images to the global coordinate system based on the cumulative affine transformation matrix, and determining the global canvas size and offset, specifically involves: For the i-th vertical stitching image Through the cumulative affine transformation matrix Transform its four corner points to the global coordinate system; traverse all vertically stitched images to obtain the minimum x-coordinate of the corner points in the global coordinate system. Minimum y-coordinate Maximum x-coordinate Maximum ordinate Then, expand outwards by a preset extension pixel d to obtain the boundary coordinates of the global canvas. The global canvas size, obtained based on boundary coordinates, is: canvas width. Canvas height Then, based on the bottom left boundary point of the global canvas, confirm the horizontal axis offset. Vertical axis offset .

7. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 6, characterized in that, The process of constructing a perspective transformation matrix based on offset and cumulative affine transformation matrix, projecting other vertically stitched images onto the global canvas at once, and performing weighted average fusion on overlapping areas to obtain the stitched complete workpiece surface image is as follows: floating-point canvas As a global canvas, the fusion weight matrix weight is initialized to 0; Each vertically stitched image is projected onto the global canvas through a perspective transformation. The perspective transformation matrix is ​​a cumulative affine transformation matrix with an offset, expressed by the formula: ； in, The i-th vertical stitching image To perspective transformation projection diagram The perspective transformation matrix; For each perspective transformation projection image Generate a binary mask: non-zero pixel regions are 1, and zero pixel regions are 0; use the binary mask as... Fusion weights ; The pixel values ​​of each perspective transformation projection image are accumulated into the floating-point canvas, and the fusion weights of each perspective transformation projection image are accumulated into the weight matrix. The pixel values ​​of each point in the accumulated floating-point canvas are divided by the accumulated weights of each point in the weight matrix to obtain the complete workpiece surface image after weighted average fusion.

8. The method for rapid stitching and acquisition of large-size aerospace structural image data according to claim 1, characterized in that, The process of detecting small-sized targets on the workpiece surface based on vertical stitched image sequences, acquiring large-sized targets on the workpiece surface based on complete workpiece surface images, and statistically analyzing and storing the categories, locations, and quantities of detected targets of different sizes is as follows: For each vertically stitched image , generate A square sliding window with a width equal to the side length slides vertically at a preset overlap ratio. Perform target detection inference on the image information within each sliding window and mark the target detection box; Restore the target detection bounding box to the vertical mosaic coordinate system. Then, it is mapped to the global coordinate system through the cumulative affine transformation matrix; Collect all mapped target detection boxes and perform non-maximum suppression in batches in the global coordinate system to obtain the detection results of small-scale targets; Full-image target detection inference is performed on the complete workpiece surface image to obtain the detection results of large-scale targets; The detection results of targets at different scales are combined and plotted on the complete workpiece surface image, and the number and location of each type of target are counted.

Images