An unmanned aerial vehicle aerial road image stitching method based on semantic mask guidance and graph cut optimization
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XINJIANG UNIVERSITY
- Filing Date
- 2026-03-10
- Publication Date
- 2026-06-09
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and UAV remote sensing image processing technology, specifically relating to a method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization. Background Technology
[0002] With the rapid development of drone technology, low-altitude remote sensing using drones has been widely applied in fields such as intelligent highway inspection, traffic accident investigation, and road surface defect detection. Because the field of view of a single drone image is limited, image stitching technology is typically used to combine multiple image sequences into a long-range panoramic image for macroscopic observation and analysis.
[0003] However, in practical highway scene stitching, two major technical challenges are faced: First, highway scenes are typical dynamic scenes, containing a large number of moving vehicles and their accompanying shadows. Traditional image stitching algorithms usually assume that the scene is static. When moving targets are present, the stitching results often exhibit "ghosting," double images, or object truncation, severely affecting the accuracy of subsequent tasks such as crack detection. Second, highway scenes have significant geometric structural features (such as straight lane lines and guardrails). Existing mesh deformation stitching methods such as APAP (As-Projective-As-Possible, a non-rigid registration algorithm for image stitching, mainly used to solve the local alignment problem caused by traditional homography transformation image stitching) can align images well, but they are prone to non-rigid distortion in areas lacking texture or with large parallax, causing originally straight road markings to become curved or distorted.
[0004] In existing technologies, traditional stitching mainly relies on color or gradient differences, which cannot "understand" and avoid vehicles at the semantic level. Therefore, there is an urgent need for a stitching method that can actively avoid dynamic vehicles and shadows while maintaining the straight structure of the road. Summary of the Invention
[0005] The purpose of this invention is to provide a method for stitching road images taken by drones based on semantic masking guidance and graph cut optimization, which aims to solve the problems of "ghosting" residue and road geometry distortion caused by stitching in dynamic highway scenes.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A method for stitching road images captured by UAVs based on semantic mask guidance and graph cut optimization includes the following steps:
[0008] S1: Acquire a sequence of road images with overlapping areas taken from above by a drone, and perform image preprocessing;
[0009] S2: The pre-trained instance shadow detection model (SSISv2) is used to extract and infer semantic features for each frame of road image, accurately identify and segment vehicle instances and their accompanying shadows on the road surface, and generate a binary semantic mask to identify dynamic interference areas.
[0010] S3: Extract sparse point features (SIFT) and edge line structure features (LSD) from neighboring images, and combine them with the RANSAC algorithm to calculate the initial registration relationship;
[0011] S4: Construct a mesh deformation model based on line-point consistency (LPC) constraints. By jointly optimizing the reprojection error of point features and the geometric structure preservation constraint of line features, a high-precision mesh deformation field is calculated.
[0012] S5: Perform synchronous geometric transformation on the aerial image and the binary semantic mask generated in step S2 using the mesh deformation field to ensure that the mask and the image remain spatially aligned after deformation;
[0013] S6: Construct an energy loss function based on graph cut to map the transformed binary semantic mask into a high-cost potential energy term in the energy field, giving the dynamic interference target extremely high traversal value in the overlapping region;
[0014] S7: Use the maximum flow minimum cut algorithm to globally optimize the energy loss function and find the best stitching line that minimizes the path cost and actively avoids vehicles and shadow areas.
[0015] S8: Based on the optimal stitching line, perform multi-band image fusion on the overlapping area to output a panoramic road image that is de-vehicled, retains real road surface texture details, and maintains the geometric structure.
[0016] Further, in step S2, the instance shadow detection model adopts an SSISv2 network containing a bidirectional relation learning module. The generation of the binary semantic mask specifically involves: uniformly marking the pixel values of the detected vehicle area and shadow area as the foreground (high potential energy area), marking the pixel values of the road background area as the background, and generating a binary mask image with the same size as the original image.
[0017] Furthermore, in step S4, the energy function of the mesh deformation model includes a line structure preservation term, which is used to constrain the deformed mesh so that the extracted line segment features maintain collinearity and straightness, thereby preventing unnatural distortion of lane lines, guardrails and curbs at the splicing points.
[0018] Furthermore, in step S6, in the definition of the data item based on the graph cut energy loss function, when a pixel is located in the foreground region of the transformed binary semantic mask, its data item cost value is set to a preset maximum penalty value, forcing the optimal stitching path to bypass the region; when a pixel is located in the background region, its data item cost value is determined by the color difference and structural difference between the reference image and the target image.
[0019] The present invention has the following beneficial effects:
[0020] This invention generates high-precision semantic masks by introducing the SSISv2 instance shadow detection model and integrates the mask information into the graph cut energy function. Unlike traditional stitching methods that rely solely on color differences, this invention can "actively" identify vehicles and shadows and force the stitching lines to bypass these dynamic areas. This completely eliminates ghosting, truncation, and double-image phenomena caused by moving vehicles in the stitching result, achieving automated removal of road surface interference targets while preserving the true road surface texture details.
[0021] To address the issue that long, straight structures such as lane lines and guardrails in highway scenes are prone to bending during mesh deformation, this invention employs a registration algorithm based on Line-Point Consistency (LPC). By introducing line feature constraints, it ensures that the straight lines and geometric shapes of the road remain straight and natural after stitching, even under conditions of high parallax or low-altitude shooting, thus avoiding the line distortion and misalignment common in traditional methods.
[0022] This invention proposes a strategy of "synchronous geometric transformation of image and mask," which utilizes the same high-precision LPC deformation field to simultaneously apply to the original image and the semantic mask. This design effectively solves the problem of spatial mismatch between semantic and texture information during non-rigid deformation, ensuring that the mask can still accurately cover the vehicle area after complex mesh deformation. This guides the graph cut algorithm for precise stitching, improving the algorithm's robustness and stitching accuracy. Attached Figure Description
[0023] Figure 1 This is a flowchart of a UAV aerial road image stitching method based on semantic mask guidance and graph cut optimization according to the present invention;
[0024] Figure 2 This is a schematic diagram of the vehicle and shadow mask generation in this invention.
[0025] Figure 3 This is a schematic diagram of point feature extraction and matching in this invention.
[0026] Figure 4 This is a schematic diagram of line feature extraction in this invention.
[0027] Figure 5 This is a schematic diagram of the splicing result in this invention. Detailed Implementation
[0028] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0029] like Figure 1 As shown, the present invention provides a method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization, which specifically includes the following steps:
[0030] Step S1: Collect a sequence of road images taken by the drone from above as input. Necessary denoising preprocessing is performed on the image, using a Gaussian Smoothing Filter to perform a convolution operation on the image to suppress high-frequency sensor noise. Let the Gaussian kernel be... The denoised image This is to improve the accuracy of subsequent feature extraction.
[0031] Step S2: Utilize the deep learning-based SSISv2 (Single-Stage Instance Shadow Detection) model to analyze the image. To reason.
[0032] The input to the SSISv2 model is The RGB image is processed by a backbone network for feature extraction and a bidirectional relationship learning module, and the output is a probability map of the same size. ,in Represents pixels Confidence score indicating the vehicle or shadow. Generate a binary semantic mask based on the detection results. Set a threshold. (For example =0.5), the mask generation formula is as follows:
[0033]
[0034] In the masked image, detected vehicle pixel regions and shadow pixel regions are marked as foreground (value set to 1), while road surface and other background regions are marked as background (value set to 0). The process is as follows: Figure 2 As shown.
[0035] Step S3: For two adjacent images (reference image) and target image Feature extraction and matching are performed, specifically including:
[0036] Keypoint features: Keypoints are extracted using the SIFT (Scale-Invariant Feature Transform) algorithm, and mismatches are removed using the RANSAC (Random Sample Consensus) algorithm to obtain a set of high-confidence matching point pairs. ,like Figure 3 As shown.
[0037] Line features: The LSD (Line Segment Detector) algorithm is used to detect line segment features in the image, resulting in a set of line segments. These line segments typically correspond to road markings with strict geometric shapes, such as lane lines and guardrails. Figure 4 As shown.
[0038] Step S4: To address the alignment problem caused by parallax and preserve the road geometry, this example introduces the Line-Point Consistency (LPC) algorithm to construct a mesh deformation model. The target image is divided into... For a uniform grid, define the total energy function. as follows:
[0039]
[0040] in: (Point Alignment Term): Used to constrain the deformed mesh so that feature points in the target image... The corresponding points in the transformed and reference images Coordinate alignment ensures accurate registration of image content.
[0041] (Line Structure Preservation Term): Introduces line feature constraints, requiring the deformed mesh to maintain the collinearity and straightness of the extracted line segments. That is, for sampling points belonging to the same lane line, the deformed mesh should still fit on a straight line to prevent unnatural bending or twisting of the lane lines at the splicing points.
[0042] (Smoothing Term): Used to constrain the deformation amplitude of mesh vertices, maintaining a smooth transition in non-textured areas.
[0043] and These are the weighting coefficients for the line constraint term and the smoothing term, respectively.
[0044] By minimizing this energy function, the optimal mesh vertex displacement vector is obtained, leading to a high-precision mesh deformation field. .
[0045] Step S5: Utilize the mesh deformation field calculated in step S4 For the target image The registered image is obtained by performing deformation using bilinear interpolation. .
[0046] Using the exact same mesh deformation field The semantic mask generated in step S2 The nearest neighbor interpolation method is used for synchronous deformation to obtain the transformed mask. .
[0047] This synchronous transformation mechanism ensures that the vehicle position in the mask corresponds strictly to the vehicle position in the registered image, avoiding semantic misalignment caused by geometric transformation.
[0048] Step S6: Construct the graph cut energy function on the reference image With the deformed target image Overlapping areas Internally, an energy loss function based on graph cut is constructed. In order to find the best suture line:
[0049]
[0050] in, For a collection of suture labels, It is a set of neighboring pixel pairs.
[0051] Data Items The specific definition of the data item reflects the pixel The cost assigned to a label (reference image or target image). This embodiment combines semantic masking and image differences to construct data items:
[0052]
[0053] For a preset maximum penalty value (e.g.) ), used to force the stitches to avoid the masked area.
[0054] , indicating color difference.
[0055] , representing gradient structure differences, is used to find the least noticeable splicing marks in a background area (such as road surface texture).
[0056] Smoothing Term The specific definition of the smoothing term is used to constrain the continuity of adjacent pixel labels and prevent discontinuities in the stitching line. In this embodiment, it is defined as follows:
[0057]
[0058] This item indicates that when the color difference between adjacent pixels is small, the cost of cutting them is high (i.e., the stitching line tends to pass through edges with large color differences, but in this scenario, it is usually guided by the data item to pass through a flat road surface).
[0059] Step S7: Construct a weighted graph , where the node set Includes all pixels and two terminal nodes (source node) Representative reference image, confluence (Represents the target image).
[0060] The weights of the edges in the graph are set according to the energy function defined in step S6: the weights of the edges (t-links) between pixel nodes and terminal nodes are determined by the data items. The weights of the edges (n-links) between adjacent pixel nodes are determined by the smoothing term. Decide.
[0061] The minimum cut of the graph is found using the Max-Flow Min-Cut algorithm. The minimum cut corresponds to the energy function. The global minimum. This is because the vehicle and the shaded area are endowed with extremely high potential energy. The minimum cut path will automatically avoid all vehicles and shadow areas and travel along the path with the least difference in background texture.
[0062] Step S8: Based on the optimal suture label obtained in step S7 Multi-band blending algorithm is used to eliminate splicing traces.
[0063] S81: Reference image and the deformed target image Construct the Laplacian Pyramid separately. and $.
[0064] S82: Generate a weighted graph based on the optimal suture line. And construct the Gaussian Pyramid on it. .
[0065] S83: On each level of the pyramid Perform weighted fusion:
[0066]
[0067] S84: Reconstruct the final panoramic image using the inverse Laplacian pyramid transformation.
[0068] In the final output panoramic image, dynamic vehicles in overlapping areas are completely removed, preserving realistic road surface texture details, and lane lines and other geometric structures remain straight and natural without significant distortion. Figure 4 As shown.
[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization, characterized in that, Includes the following steps: Step S1: Obtain a sequence of road images with overlapping areas taken from above by the UAV, and perform preprocessing; Step S2: Use a pre-trained instance shadow detection model to extract and infer semantic features for each frame of road image, identify vehicle instances on the road surface and their accompanying shadows, and generate corresponding binary semantic masks; Step S3: Extract sparse point features and edge line structure features from adjacent images, and calculate the initial registration relationship; Step S4: Construct a mesh deformation model based on line-point consistency constraints. Calculate a high-precision mesh deformation field by jointly optimizing the reprojection error of point features and the geometric structure preservation constraint of line features. Step S5: Perform synchronous geometric transformation on the aerial image and the binary semantic mask generated in step S2 using the mesh deformation field, and transform the image and mask to the reference coordinate system. Step S6: Construct an energy loss function based on graph cut, map the transformed binary semantic mask into a high-cost potential energy term in the energy field, and give the dynamic interference target a very high crossing value in the overlapping region; Step S7: Use the maximum flow minimum cut algorithm to globally optimize the energy loss function and find an optimal stitching line that actively avoids vehicles and shadow areas; Step S8: Based on the optimal stitching line, perform multi-band image fusion on the overlapping area to output a panoramic road image that is de-vehicled, retains real road surface texture details, and maintains the geometric structure.
2. The method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, In step S2, the instance shadow detection model adopts the SSISv2 (Single-Stage Instance Shadow Detection) network model; The specific steps for generating the corresponding binary semantic mask are as follows: setting the pixel values of the detected vehicle region and shadow region to 1, and setting the pixel values of the road background region to 0, to generate a binary mask image with the same size as the original image. .
3. The method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, Step S3 specifically includes: The SIFT (Scale-Invariant Feature Transform) algorithm is used to extract key point features between adjacent images, and the RANSAC (Random Sample Consensus) algorithm is used to remove mismatched points. The global homography matrix is calculated as the initial registration parameter. The LSD (Line Segment Detector) algorithm is used to extract line segment features from the image, which are then used for geometric constraints in subsequent mesh deformation.
4. The method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, In step S4, the construction of the mesh deformation model based on line-point consistency constraints, its total energy function Defined as: in, This is a point feature alignment term used to constrain the deformed mesh so that the coordinates of matching point pairs coincide. The line structure preservation term is used to constrain the deformed mesh so that the extracted line segment features maintain collinearity and straightness, preventing geometric distortion of lane lines and guardrails; This is a mesh smoothing term used to constrain the deformation amplitude of mesh vertices and maintain a smooth transition in non-textured areas; and These are the weighting coefficients for the line constraint term and the smoothing term, respectively.
5. The method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, In step S5, the synchronous geometric transformation specifically includes: Using the mesh vertex displacement vectors calculated in step S4, the aerial image is resampled and deformed using the bilinear interpolation method; Using the same grid vertex displacement vector, the binary semantic mask is resampled and deformed by the nearest neighbor interpolation method to maintain the sharpness of the mask edges and ensure that the mask and the deformed image are strictly aligned in space.
6. The method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, In step S6, the energy loss function based on graph cut... Defined as: in, For a collection of suture labels, The set of pixels in the overlapping region. It is a set of neighboring pixel pairs; This is a smoothing term used to penalize discontinuities in adjacent pixel labels, guiding the stitching line through the region with the least texture difference; For the data item, combined with the aforementioned binary semantic mask, the definition is as follows: when pixel When the data item is located in the high-potential region of the transformed binary semantic mask (i.e., the mask value is 1), its data item cost is set to a preset maximum penalty value. When pixel When located in the background region (i.e., the mask value is 0), the data item value is mainly determined by the color difference between the reference image and the target image.
7. A method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 6, characterized in that, The maximum penalty value The value of is much larger than the maximum possible value of the color difference term, so as to ensure that the optimal stitching path cannot pass through the high potential energy region when using the maximum flow minimum cut algorithm.
8. A method for stitching UAV aerial road images based on semantic mask guidance and graph cut optimization according to claim 1, characterized in that, In step S8, the multi-band image fusion specifically involves: Using the optimal seam line obtained from the solution as the boundary, the overlapping area between the reference image and the target image is divided into a left region and a right region; A Laplacian pyramid was constructed, and the images on both sides of the stitching line were weighted and fused at different frequency bands. Finally, the image was reconstructed to eliminate stitching marks and preserve road surface texture details.