A method, system, storage medium, and program product for stitching building facade images based on continuous single-view UAV images.

Abstract

This application provides a method, system, storage medium, and program product for stitching building facade images based on continuous single-view imagery from a UAV, relating to the field of UAV image processing technology. The method includes: extracting features from multiple building facade images continuously acquired from a single UAV viewpoint to obtain an initial feature point set; performing corner detection on the initial feature point set to obtain a candidate corner point set, and performing line segment detection on the building facade images to obtain a line segment set; selecting a structured feature point set from the candidate corner point set and line segment set based on the structural characteristics of the building facade, and performing category weighting to obtain an enhanced feature point set; performing feature matching on the enhanced feature point sets of adjacent first and second adjacent images to obtain a set of matching point pairs; and performing image synthesis on the first and second adjacent images based on the set of matching point pairs to obtain a target panoramic image. This achieves the technical effect of improving the stitching quality of panoramic images of building facades.

Description

Technical Field

[0001] This application relates to the field of UAV image processing technology, and in particular to a method, system, storage medium and program product for stitching building facade images based on continuous images from a single UAV viewpoint. Background Technology

[0002] With the widespread application of low-altitude drone technology, using drones to continuously photograph building facades along preset routes and generating panoramic images of building facades with a wide field of view through image stitching technology has become an important means of monitoring the structural health of buildings and digital inspection.

[0003] In related technologies, feature point detection algorithms are typically used to stitch together continuously acquired building facade images by drones. The specific implementation process is as follows: First, the drone continuously acquires building facade images using a feature point detection algorithm, extracting feature points from each acquired facade image. Then, based on the similarity of feature vectors, the feature points of adjacent facade images are compared one by one to obtain matching point pairs. Next, the transformation matrix between the two adjacent facade images is calculated using the matching point pairs. Finally, the transformation matrix is ​​used to perform coordinate transformation and stitching on the two adjacent facade images to generate the desired panoramic image of the building facade.

[0004] However, using the above method, due to the complex texture structure of the building facade, it is difficult to accurately represent the structural features of the building facade. This makes it easy for incorrect matching point pairs to occur during the comparison of two adjacent building facade images. As a result, the building facade in the stitched panoramic image may have structural misalignment, structural distortion, and structural breakage, which in turn leads to structural distortion in the stitched panoramic image of the building facade in related technologies. Summary of the Invention

[0005] This application provides a method, system, storage medium, and program product for stitching building facade images based on single-view continuous images from unmanned aerial vehicles (UAVs), which can be used to improve the stitching quality of panoramic images of building facades.

[0006] In a first aspect, this application provides a method for stitching building facade images based on continuous single-view images from a UAV, applied to a building facade image stitching system. The method includes: acquiring multiple building facade images of a target building continuously captured from a single viewpoint by a UAV; extracting features from each of the multiple building facade images to obtain an initial feature point set; performing corner detection on the initial feature point set to obtain a candidate corner point set; and performing line segment detection on each building facade image to obtain a line segment set, and selecting a structured feature point set from the candidate corner point set and the line segment set based on the structural characteristics of the target building's facade; performing category weight enhancement on the structured feature point set to obtain an enhanced feature point set; determining adjacent first and second adjacent images from the multiple building facade images, and determining a first feature point set for the first adjacent image and a second feature point set for the second adjacent image from the enhanced feature point set; performing feature matching between the first and second feature point sets to obtain a set of matched point pairs; and synthesizing the first and second adjacent images based on the set of matched point pairs to obtain a target panoramic image.

[0007] By employing the aforementioned technical solution, feature extraction is performed on each building facade image continuously acquired from a single viewpoint by a UAV to obtain an initial feature point set. Corner detection is then performed on the initial feature point set, and line segment detection is performed on each building facade image. Based on the structural characteristics of the target building's facade, a structured feature point set with clear architectural geometric significance is selected from the candidate corner point set and line segment set. This structured feature point set is then weighted by category to obtain an enhanced feature point set, allowing key architectural structural feature points to play a dominant role in subsequent matching. Feature matching and image synthesis are then performed based on the enhanced feature point set, thereby ensuring the continuity and integrity of key geometric structures such as windows, floor lines, and drainage pipes on the building facade during the stitching process. This solves the technical problem of structural distortion in stitched panoramic images of building facades in related technologies, achieving the technical effect of improving the stitching quality of panoramic images of building facades.

[0008] Optionally, each candidate corner point in the candidate corner point set is traversed sequentially, and the following operations are performed on the currently traversed candidate corner point: Horizontal line segments are selected from the line segment set whose first angle with the horizontal direction is less than a first preset angle threshold, and whose first shortest distance to the target corner point is less than a preset distance threshold; the target corner point is the currently traversed candidate corner point. Vertical line segments are selected from the line segment set whose second angle with the vertical direction is less than a second preset angle threshold, and whose second shortest distance to the target corner point is less than a preset distance threshold; the first preset angle threshold is less than the second preset angle threshold. When both horizontal and vertical line segments are selected simultaneously, the target corner point is determined to be the planar window corner point formed by the horizontal and vertical line segments. The planar window corner point is added to the window corner point set until each candidate corner point is traversed, and the structured feature point set includes the window corner point set.

[0009] By adopting the above technical solution, and taking advantage of the structural characteristics that windows in building facades typically present rectangular or near-rectangular outlines, the system determines whether horizontal and vertical line segments that satisfy both angle and distance conditions exist simultaneously in the corresponding neighborhood of each candidate corner point in the candidate corner point set. This allows for the accurate identification of planar window corner points located at the corners of the window's rectangular outline, enabling the selected set of structured feature points to accurately represent the window structure of the building facade, thereby improving the accuracy of subsequent feature matching and the integrity of the building structure.

[0010] Optionally, based on the structural characteristics of the target building's facade, a set of structured feature points is selected from the candidate corner point set and the line segment set. This includes: extracting an approximate horizontal line segment set and an approximate vertical line segment set from the line segment set; pairwise combining the approximate horizontal line segment set and the approximate vertical line segment set to obtain multiple approximate line segment combination sets; sequentially traversing each approximate line segment combination in the multiple approximate line segment combination sets, and performing the following operation on the currently traversed approximate line segment combination: determining whether the first extension line of the target approximate horizontal line segment and the second extension line of the target approximate vertical line segment are adjacent. The current traversed approximate line segment combination includes the target approximate horizontal line segment and the target approximate vertical line segment. When determining that the first extension line intersects with the second extension line, it is determined whether the target intersection point of the first extension line and the second extension line is located within the effective area of ​​the current building facade image. When it is determined that the target intersection point is located within the effective area, the coordinates of the target intersection point are determined. The target intersection point is determined as the linear floor line intersection point, and the linear floor line intersection point is added to the floor line intersection point set until each approximate line segment combination is traversed. The structured feature point set includes the floor line intersection point set.

[0011] By adopting the above technical solution, approximate horizontal line segment sets and approximate vertical line segment sets are extracted from the line segment set and combined in pairs. The intersection point of the first extension line of the target approximate horizontal line segment and the second extension line of the target approximate vertical line segment in each approximate line segment combination is calculated. When the target intersection point is located within the effective area of ​​the current building facade image, the intersection point is determined as the linear floor line intersection point. This accurately identifies the floor line intersection points of the corresponding building structure corners or boundaries, enabling the structured feature point set to effectively characterize the floor boundary structure of the building facade, and further improving the continuity and alignment accuracy of horizontal and vertical structures such as floor lines during the splicing process.

[0012] Optionally, based on the structural characteristics of the target building's facade, a set of structured feature points is selected from the candidate corner point set and the line segment set. This includes: selecting a set of candidate pipe segments that meet preset pipe segment conditions from the line segment set, where the preset pipe segment conditions are that the line segment length is greater than a preset length threshold and the line segment width is less than a preset width threshold; extracting the two endpoints of each candidate pipe segment in the candidate pipe segment set to obtain a set of pipe endpoints; determining whether multiple candidate pipe segments intersect at the same location based on the set of pipe endpoints; when it is determined that multiple candidate pipe segments intersect at the same location, clustering the multiple endpoints intersecting at the same location among the multiple candidate pipe segments to obtain branch nodes; identifying the branch nodes and the endpoints in the set of pipe endpoints that did not participate in the clustering process as drainage pipe nodes, and adding the drainage pipe nodes to the set of drainage pipe nodes. The set of structured feature points includes the set of drainage pipe nodes.

[0013] By adopting the above technical solution, candidate pipe segments that meet the conditions of having a length greater than a preset length threshold and a width less than a preset width threshold are selected from the set of line segments. The endpoints of the candidate pipe segments are extracted, and it is determined whether multiple candidate pipe segments intersect at the same position. When there is an intersection, branch nodes are obtained through clustering. Finally, the branch nodes and the endpoints that did not participate in the clustering process are identified as drainage pipe nodes. This accurately identifies the endpoints and branch points in the drainage pipe structure of the building facade, so that the structured feature point set can completely represent the drainage pipe structure of the building facade, and improves the continuity and splicing accuracy of linear structures such as drainage pipes during the splicing process.

[0014] Optionally, the structured feature point set is enhanced by class weighting to obtain an enhanced feature point set, including: sequentially traversing each structured feature point in the structured feature point set, and performing the following operations on the currently traversed structured feature point: determining the target class to which the target structured feature point belongs, and determining the class weight coefficient of the target structured feature point based on the target class, wherein the target structured feature point is the currently traversed structured feature point; determining the corner response value and local feature point density of the target structured feature point; determining the weight value of the target structured feature point based on the class weight coefficient, corner response value, local feature point density, and preset balance coefficient; directly After traversing each structured feature point to obtain the weight value of each structured feature point; determine the Euclidean distance between any two structured feature points in the set of structured feature points; based on the Euclidean distance, if there exists a first structured feature point and a second structured feature point in the set of structured feature points whose Euclidean distance is less than a preset minimum distance threshold, compare the weight values ​​of the first structured feature point and the second structured feature point to obtain the weight comparison result; based on the weight comparison result, remove the first or second structured feature point that does not meet the preset weight retention condition from the set of structured feature points to obtain the enhanced feature point set.

[0015] By adopting the above technical solution, the category weight coefficient is determined according to the target category to which each structured feature point belongs in the structured feature point set. The weight value of each structured feature point is calculated by combining the corner response value and the local feature point density. This allows structured feature points of different categories and different saliences to obtain differentiated weights, highlighting the dominant role of key building structure feature points. At the same time, by comparing the weights of structured feature points whose Euclidean distance is less than the preset minimum distance threshold and removing structured feature points with low weights, redundant feature points in overly dense areas are effectively suppressed, and the spatial distribution uniformity of feature points is optimized, thereby improving the efficiency and accuracy of subsequent feature matching.

[0016] Optionally, the first adjacent image and the second adjacent image are synthesized based on the set of matching point pairs to obtain the target panoramic image. This includes: performing the following iterative operations on the set of matching point pairs according to a preset number of iterations: randomly selecting a preset number of non-collinear matching point pairs from the set of matching point pairs and determining the candidate homography matrix of the non-collinear matching point pairs; determining the projection error of other matching point pairs in the set of matching point pairs based on the candidate homography matrix, wherein the other matching point pairs are the remaining matching point pairs in the set of matching point pairs excluding the non-collinear matching point pairs; and determining the other matching point pairs whose projection error is less than a preset error threshold as... The process involves identifying inlier points and determining their number; iterating for a preset number of iterations to obtain multiple candidate homography matrices and the number of inlier points corresponding to each candidate homography matrix; determining the candidate homography matrix that satisfies the preset number of inlier points from among the multiple candidate homography matrices as the target homography matrix based on the number of inlier points; determining all inlier points corresponding to the target homography matrix as the target inlier point set; re-estimating and optimizing the target homography matrix based on the target inlier point set to obtain the optimized homography matrix; and stitching the first adjacent image and the second adjacent image together based on the optimized homography matrix to obtain the target panoramic image.

[0017] By adopting the above technical solution, the matching point pair set is iteratively randomly sampled and verified according to a preset number of iterations. In each iteration, non-collinear matching point pairs are randomly selected to calculate the candidate homography matrix and count the number of interior points. The optimal set of interior points is effectively selected from the set of matching point pairs containing mismatches, and the mismatched point pairs are eliminated. Then, the selected target interior point set is used to re-estimate and optimize the target homography matrix to obtain a higher precision optimized homography matrix, thereby ensuring the accuracy of image transformation and improving the alignment accuracy and integrity of the building structure in the stitched panoramic image of the building facade.

[0018] Optionally, the first adjacent image and the second adjacent image are stitched together according to the optimized homography matrix to obtain the target panoramic image, including: performing perspective transformation on the second adjacent image according to the optimized homography matrix to obtain a transformed image; creating a blank canvas and placing the first adjacent image in the blank canvas according to the original coordinates to obtain a first canvas image; determining the transformation coordinates of the transformed image according to the optimized homography matrix and placing the transformed image in the blank canvas according to the transformation coordinates to obtain a second canvas image; determining the overlapping area between the first canvas image and the second canvas image, and the first non-overlapping area of ​​the first canvas image, based on the positional relationship between the first canvas image and the second canvas image in the blank canvas. The overlapping region and the second non-overlapping region of the second canvas image; constructing a first Gaussian pyramid for the first adjacent image, and constructing a second Gaussian pyramid for the transformed image; performing first adjacent layer difference processing on the first Gaussian pyramid to generate a first Laplacian pyramid, and performing second adjacent layer difference processing on the second Gaussian pyramid to generate a second Laplacian pyramid; constructing a bottom-level weight map based on the overlapping region, the first non-overlapping region, and the second non-overlapping region; performing multi-scale decomposition processing on the bottom-level weight map to obtain a weight pyramid, and performing multi-resolution fusion processing on the first Laplacian pyramid and the second Laplacian pyramid based on the weight pyramid to obtain a target panoramic image.

[0019] By adopting the above technical solution, the second adjacent image is transformed using an optimized homography matrix, and the first adjacent image and the transformed image are placed in a blank canvas to determine the overlapping area, the first non-overlapping area, and the second non-overlapping area. Then, the first Gaussian pyramid and the second Gaussian pyramid are constructed respectively, and the first Laplacian pyramid and the second Laplacian pyramid are generated through adjacent layer difference processing. Then, the bottom layer weight map is constructed and multi-scale decomposition is performed to obtain the weight pyramid. Finally, the first Laplacian pyramid and the second Laplacian pyramid are fused according to the weight pyramid, so as to realize weighted fusion in different frequency bands, thereby preserving high-frequency detail information and smoothing low-frequency transition, effectively eliminating brightness differences and ghosting phenomena at the stitching seam, and generating a seamless target panoramic image.

[0020] In a second aspect, embodiments of this application provide a building facade image stitching system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the building facade image stitching system to perform the method as described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer-readable storage medium including program instructions that, when executed on a building facade image stitching system, cause the building facade image stitching system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a building facade image stitching system, cause the building facade image stitching system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By selecting and weighting feature points for three typical structural elements of building facades—window corners, floor line intersections, and drainage pipe nodes—the extracted feature points accurately correspond to the key geometric elements of the building facade, thereby improving the repeatability and representativeness of feature points across continuous building facade images.

[0025] 2. By filtering structured feature points, the number of feature points involved in matching is greatly reduced, while the ambiguous matching of repeated texture areas on the building facade is effectively reduced, thereby improving the accuracy and computational efficiency of feature matching.

[0026] 3. By taking key architectural structural feature points such as window corners, floor line intersections, and drainage pipe nodes as the protected objects for stitching, higher weights are assigned during the category weighting enhancement stage, and they participate in the constraint as interior points in the subsequent homography matrix estimation, ensuring the continuity and non-misalignment of key geometric structures such as windows, floor lines, and drainage pipes on the building facade in the stitched target panoramic image. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a method for stitching together building facade images based on continuous single-view images from a UAV, as described in this application.

[0028] Figure 2 This is a schematic diagram of the physical device structure of a building facade image stitching system in the embodiments of this application. Detailed Implementation

[0029] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0031] This application provides a method for stitching building facade images based on continuous single-view UAV imagery. (See reference...) Figure 1 , Figure 1 This is a flowchart illustrating a method for stitching together building facade images based on continuous single-view UAV images, as described in this application, including the following steps:

[0032] Step S101: Acquire multiple exterior facade images of the target building continuously collected by the UAV from a single perspective, and extract features from each exterior facade image in the multiple exterior facade images to obtain an initial feature point set.

[0033] Step S102: Perform corner detection on the initial feature point set to obtain a candidate corner point set; and perform line segment detection on each building facade image to obtain a line segment set. Then, select a structured feature point set from the candidate corner point set and the line segment set according to the building facade structural characteristics of the target building.

[0034] Step S103: Perform class weight enhancement on the structured feature point set to obtain the enhanced feature point set;

[0035] Step S104: Determine the first adjacent image and the second adjacent image from multiple building facade images, and determine the first feature point set of the first adjacent image and the second feature point set of the second adjacent image from the enhanced feature point set;

[0036] Step S105: Perform feature matching between the first feature point set and the second feature point set to obtain a set of matching point pairs;

[0037] Step S106: Combine the first adjacent image and the second adjacent image according to the set of matching point pairs to obtain the target panoramic image.

[0038] Among them, "single-view continuous acquisition by UAV" refers to the continuous acquisition of the exterior facade of the target building by the UAV along a preset flight path at a fixed shooting angle, with a certain proportion of overlapping area between adjacent images of the building facade; "target building" refers to the building whose facade images need to be stitched together; "building facade image" refers to a single image of the exterior facade of the target building taken by the UAV; "initial feature point set" refers to the set of feature points obtained after multi-scale feature extraction of each building facade image and after density control and low response point removal, and each feature point in the initial feature point set contains information such as position, scale, main direction and descriptor; "candidate corner point set" refers to the set of corner points obtained after corner detection of the initial feature point set; "line segment set" refers to the set of line segments obtained after line segment detection of each building facade image; "building facade structural characteristics" refers to typical geometric structural characteristics of the building facade, such as windows having rectangular outlines, floor lines being horizontal or vertical long straight segments, and drainage pipes being slender vertical straight segments.

[0039] Among them, the structured feature point set refers to the set of feature points with clear architectural geometric meaning selected from the candidate corner point set and line segment set based on the structural characteristics of the building facade. The structured feature point set includes the window corner point set, the floor line intersection point set, and the drainage pipe node set; category weight enhancement refers to assigning a corresponding category weight coefficient to each structured feature point in the structured feature point set according to its category, and calculating the weight value by combining the corner point response value and the local feature point density, while suppressing and eliminating structured feature points that are too close; the enhanced feature point set refers to the optimized feature point set obtained after category weight enhancement of the structured feature point set; the first adjacent image and the... Two adjacent images refer to two adjacent building facade images determined from multiple building facade images in the order of acquisition, where the first adjacent image is used as the reference image and the second adjacent image is used as the target image; the first feature point set and the second feature point set refer to the sets of feature points belonging to the first and second adjacent images determined from the enhanced feature point set, respectively; the matching point pair set refers to the set of effective matching point pairs obtained after feature matching of the first feature point set and the second feature point set; the target panoramic image refers to the seamlessly stitched panoramic image of the building facade obtained by image synthesis of the first adjacent image and the second adjacent image based on the matching point pair set.

[0040] In the above embodiments, candidate feature points are rapidly extracted from each building facade image acquired by the UAV along a continuous trajectory from a single perspective: an improved ORB or SIFT algorithm is used for feature point extraction in linear scale space, and the KAZE algorithm is used for feature point extraction in nonlinear scale space. To balance speed and accuracy, a fast explicit diffusion algorithm can be used to accelerate the construction of the nonlinear scale space. The nonlinear diffusion filtering process solved by this fast explicit diffusion algorithm can be described by the following partial differential equation:

[0041]

[0042] in, This is an image brightness function, representing the brightness values ​​of the image at different scales; Image brightness function For scale parameters The partial derivatives; This is a scale parameter that controls the degree of diffusion; The transfer function controls the pixel coordinates. Location, scale Diffusion rate and conduction function Smaller values ​​are used in edge regions to suppress cross-edge diffusion, and larger values ​​are used in flat regions to allow smooth diffusion. For divergence operators; Image brightness function The spatial gradient. After feature point extraction, density control and low-response point removal are performed on all initially extracted feature points to obtain a uniformly distributed, high-quality feature point set for subsequent screening. Within each building facade image, the Adaptive Non-Maximum Suppression (ANMS) algorithm or directly retaining several feature points with the highest response values ​​is used to control the density and uniformity of feature point distribution. Specifically, for the entire building facade image, its feature point set is: Calculate each feature point. response value (Such as Harris corner response, DoG response, or SIFT contrast value). Retain the value with the highest response within the image. One point, It can be dynamically adjusted according to the richness of image texture (texture-rich areas) Larger areas with poor texture (Smaller). Simultaneously, set a global threshold. Points with excessively low response values ​​are removed. The feature point response value can be calculated using the following formula (taking Harris corner points as an example):

[0043]

[0044] in, For the first Harris corner response values ​​of each feature point; This is the autocorrelation matrix at that feature point; Autocorrelation matrix The determinant of; Autocorrelation matrix The trace (i.e., the sum of the elements on the diagonal of the matrix); Autocorrelation matrix The trace, i.e., the autocorrelation matrix The sum of the diagonal elements; This is an empirical constant (usually taken as 0.04~0.06). The output feature point set. It contains information such as the location, scale, main direction, and descriptor of each feature point, providing a foundation for subsequent feature point selection based on building structure.

[0045] In the above embodiments, based on the initial feature point set extracted in the aforementioned embodiments, feature points with clear geometric significance for the building facade are selected, including corner points of planar windows, intersections of linear floor lines, and nodes of linear drainage pipes. These structural edge feature points are then enhanced, while simultaneously making the geometric distribution of feature points more uniform. Specifically, from the initial feature point set... In the process, obtain the candidate corner point set. The LSD algorithm is used to process the image. Perform line segment detection to obtain a line segment set. The process involves obtaining candidate corner point sets and line segment sets. Corner point detection employs an improved Shi-Tomasi algorithm, calculating the response value for each pixel. :

[0046]

[0047] in, pixel position Shi-Tomasi corner response value at the location; and pixel position Calculate the autocorrelation matrix Two eigenvalues; Indicates taking and The smaller value in the range. Set a threshold. ,reserve The pixels are used as candidate corner points to form a set. Line segment detection uses the LSD (LineSegment Detector) algorithm to extract a set of line segments from the image. Each straight line segment From the corresponding endpoint coordinates and This is indicated by... Based on the structural characteristics of the building facade, candidate corner point sets and line segment sets are selected... Three types of key feature points were selected from the data. For the first The set of all candidate feature points extracted from a building facade image; To The feature point subset obtained after adaptive nonmaximum suppression and response value filtering (superscript) This indicates that the data has undergone max-suppression and number-control processing. ; The number of feature points to retain (usually 500-2000, which can be dynamically adjusted according to the richness of the image texture); This is the global response value threshold (usually an empirical value, used to eliminate weak feature points with excessively low response values). The threshold value for Shi-Tomasi corner response; This is the final set of feature points (the superscript * indicates the final result after complete filtering).

[0048] In the above embodiments, the filtered feature points are used to analyze adjacent images. and feature point set and Perform high-precision matching. For Each feature point in Its feature descriptor is denoted as .exist Search for the two feature points that are closest to the Euclidean distance of the descriptor. and The corresponding distances are respectively and ( The formula for calculating Euclidean distance is:

[0049]

[0050] in, To describe the sub-dimension (SIFT is 128, SURF is 64, etc.) For feature points With feature points The Euclidean distance between them; For feature points Feature descriptors; For feature points Feature descriptors; To describe the subdimension (SIFT is 128, SURF is 64, etc.); For feature points Feature descriptors The One component; For feature points Feature descriptors The Each component. Lowe's ratio test is used for initial matching. For feature points... If its nearest neighbor distance Distance to the next nearest neighbor The ratio is less than a preset threshold Then it is believed and This constitutes a valid matching point. Mathematically, this is expressed as:

[0051]

[0052] in The value typically ranges from 0.6 to 0.8, with 0.7 being preferred in this embodiment. This criterion effectively eliminates ambiguous matches caused by repeated textures or blurring, significantly improving the matching accuracy. Performing the above operation on all feature points yields an initial set of matching point pairs. .

[0053] Through the above steps, feature extraction is performed on each building facade image continuously acquired from a single viewpoint by the UAV to obtain an initial feature point set. Corner detection and line segment detection are then performed on the initial feature point set for each building facade image. Based on the structural characteristics of the target building's facade, a structured feature point set with clear architectural geometric significance is selected from the candidate corner point set and line segment set. This structured feature point set is then weighted by category to obtain an enhanced feature point set, allowing key architectural structural feature points to play a dominant role in subsequent matching. Feature matching and image synthesis are then performed based on the enhanced feature point set, thereby ensuring the continuity and integrity of key geometric structures such as windows, floor lines, and drainage pipes on the building facade during the stitching process. This solves the technical problem of structural distortion in stitched panoramic images of building facades in related technologies, achieving the technical effect of improving the stitching quality of panoramic images of building facades.

[0054] The entity performing the above steps may be a system or device, or a controller or processor in the device or system, or a separate controller or processor, or other processing devices or processing units with similar processing functions, but is not limited to these.

[0055] In an optional embodiment, each candidate corner point in the candidate corner point set is traversed sequentially, and the following operations are performed on the currently traversed candidate corner point: Horizontal line segments with a first angle to the horizontal direction less than a first preset angle threshold and a first shortest distance to the target corner point less than a preset distance threshold are selected from the line segment set; the target corner point is the currently traversed candidate corner point; Vertical line segments with a second angle to the vertical direction less than a second preset angle threshold and a second shortest distance to the target corner point less than a preset distance threshold are selected from the line segment set; the first preset angle threshold is less than the second preset angle threshold; When both horizontal and vertical line segments are selected simultaneously, the target corner point is determined to be a planar window corner point formed by the horizontal and vertical line segments; The planar window corner point is added to the window corner point set until each candidate corner point is traversed, and the structured feature point set includes the window corner point set.

[0056] Here, candidate corner points refer to each corner point in the candidate corner point set; target corner point refers to the currently traversed candidate corner point; the first included angle refers to the angle between the line segment in the line segment set and the horizontal direction; the first preset angle threshold is the angle threshold used to determine whether the line segment is approximately horizontal, which can be 5° to 10°, etc.; the first shortest distance is the shortest distance from the target corner point to the horizontal line segment; the preset distance threshold is the distance threshold used to determine whether the target corner point and the line segment are adjacent, which can be 5 to 10 pixels, etc.; the horizontal line segment refers to the line segment in the line segment set whose first included angle with the horizontal direction is less than the first preset angle threshold and whose first shortest distance with the target corner point is less than the preset distance threshold; the second included angle refers to... The angle between the line segments in the line segment set and the vertical direction; the second preset angle threshold is the angle threshold used to determine whether a line segment is approximately vertical, which can be 10° to 15°, etc., and the second preset angle threshold is greater than the first preset angle threshold; the second shortest distance is the shortest distance from the target corner point to the vertical line segment; the vertical line segment is the line segment in the line segment set whose second angle with the vertical direction is less than the second preset angle threshold and whose second shortest distance to the target corner point is less than the preset distance threshold; the planar window corner point is the target corner point located at the corner of the window rectangle outline when both horizontal and vertical line segments that meet the conditions are selected simultaneously; the window corner point set is the set of all points determined as planar window corner points.

[0057] In the above embodiments, based on the structural characteristics of windows that typically present a rectangular or near-rectangular outline, line segment sets are utilized. The corner points of the planar window are filtered using mutually perpendicular and adjacent line segments. Specifically, for the initial candidate corner point set... Each candidate corner point (i.e., the target corner point), from the line segment set Filter horizontal line segments that simultaneously meet both angle and distance conditions. and vertical line segments This is used to determine whether a target corner point is a corner point of a planar window. The specific filtering formula is as follows:

[0058]

[0059] in, For the set of line segments The horizontal line segments selected from the data; For the set of line segments Vertical line segments selected from the data; The first angle between the horizontal line segment and the horizontal direction; It is the second angle between the vertical line segment and the vertical direction; The first preset angle threshold; The second preset angle threshold is used. It should be noted that the first preset angle threshold is smaller than the second preset angle threshold. This is because when collecting images of the building facade, the vertical edges of the building are prone to large tilting and shrinkage in the image due to the perspective effect caused by the shooting angle (such as shooting from above). The angle deviation of the horizontal edges is relatively small. Therefore, a relatively larger angle tolerance range is set for the vertical line segments to improve the fault tolerance of feature extraction. For the target corner point The first shortest distance to the horizontal line segment; For the target corner point The second shortest distance to the vertical line segment; A preset distance threshold is set (usually 5-10 pixels). Horizontal line segments that simultaneously meet the above angle and distance conditions are then selected. and vertical line segments At that time, determine the target corner point horizontal line segment With the vertical line segment The resulting planar window corner points are then added to the window corner point set. This process continues until every candidate corner point has been traversed. The final structured feature point set includes the window corner point set.

[0060] In an optional embodiment, a set of structured feature points is selected from a set of candidate corner points and a set of line segments based on the structural characteristics of the target building's facade. This includes: extracting a set of approximately horizontal line segments and a set of approximately vertical line segments from the set of line segments; combining the sets of approximately horizontal line segments and the sets of approximately vertical line segments pairwise to obtain multiple sets of approximately line segment combinations; sequentially traversing each of the multiple sets of approximately line segment combinations, and performing the following operation on the currently traversed approximately line segment combination: determining the first extension of the target approximately horizontal line segment and the second extension of the target approximately vertical line segment. Whether the lines intersect, the currently traversed approximate line segment combination includes the target approximate horizontal line segment and the target approximate vertical line segment; when it is determined that the first extension line intersects with the second extension line, it is determined whether the target intersection point of the first extension line and the second extension line is located within the effective area of ​​the current building facade image; when it is determined that the target intersection point is located within the effective area, the coordinates of the target intersection point are determined; the target intersection point is determined as the linear floor line intersection point, and the linear floor line intersection point is added to the floor line intersection point set, until each approximate line segment combination is traversed, and the structured feature point set includes the floor line intersection point set.

[0061] The approximate horizontal line segment set refers to the set of line segments extracted from the line segment set whose angle with the horizontal direction is less than a third preset angle threshold; the approximate vertical line segment set refers to the set of line segments extracted from the line segment set whose angle with the vertical direction is less than a third preset angle threshold; the third preset angle threshold is the angle threshold used to determine whether a line segment is approximately horizontal or approximately vertical, usually taken as 5° to 15°; the approximate line segment combination set refers to the set of all approximate line segment combinations obtained by pairwise combination of each approximate horizontal line segment in the approximate horizontal line segment set with each approximate vertical line segment in the approximate vertical line segment set; the approximate line segment combination refers to each combination in the approximate line segment combination set, each approximate line segment combination including a target approximate horizontal line segment and a target approximate vertical line segment. The target approximate horizontal line segment refers to the approximate horizontal line segment in the currently traversed combination of approximate line segments; the target approximate vertical line segment refers to the approximate vertical line segment in the currently traversed combination of approximate line segments; the first extension line refers to the straight line obtained by extending the target approximate horizontal line segment infinitely in both directions along its current straight line; the second extension line refers to the straight line obtained by extending the target approximate vertical line segment infinitely in both directions along its current straight line; the target intersection point refers to the intersection point of the first extension line and the second extension line; the effective area refers to the rectangular area enclosed by the image boundary of the current building facade image; the linear floor line intersection point refers to the target intersection point located within the effective area, which corresponds to the corner or boundary of the building structure; the floor line intersection point set refers to the set of all points determined as linear floor line intersection points.

[0062] In the above embodiments, floor lines are mainly represented by long straight lines, either horizontal or vertical, and the intersections of floor lines often correspond to the corners or boundaries of the building structure. Based on the structural characteristics of the target building's facade, the process of selecting a set of structured feature points from the candidate corner point set and line segment set is as follows: Extracting an approximate horizontal line segment set from the line segment set. and approximately perpendicular line segment set Specifically, for each line segment in the set of line segments, calculate the third angle between the current line segment and the horizontal direction. When the third angle is less than or equal to θ, the current line segment is assigned to the set of approximately horizontal line segments. When (90°-θ)≤the third included angle≤90°, the current line segment is assigned to the set of approximately perpendicular line segments. θ is a third preset angle threshold. For example, θ can be 5°, 10°, 15°, etc., and can also be flexibly set according to the inclination of the floor lines in the actual building facade image. For example, when θ is 10°, line segments with an angle of no more than 10° to the horizontal direction are judged as approximately horizontal line segments, and line segments with an angle of no more than 10° to the vertical direction are judged as approximately vertical line segments. The set of approximately horizontal line segments... Each set of approximate horizontal and approximate vertical line segments Each approximately vertical line segment in the set is combined in pairs to obtain multiple combinations of approximately line segments. For example, if the set of approximately horizontal line segments... It contains M approximately horizontal line segments and a set of approximately vertical line segments. If there are N approximately vertical line segments, then there are a total of M×N combinations of approximately line segments. All combinations of approximately line segments constitute the set of combinations of approximately line segments.

[0063] In the above embodiment, assuming that the currently traversed approximate line segment combination includes a target approximate horizontal line segment and a target approximate vertical line segment, the following steps are performed on the currently traversed approximate line segment combination: Step 1, extend the target approximate horizontal line segment infinitely in both directions along the direction of its line to obtain a first extension line; extend the target approximate vertical line segment infinitely in both directions along the direction of its line to obtain a second extension line. Since the directions of the target approximate horizontal line segment and the target approximate vertical line segment are approximately perpendicular, the first extension line intersects the second extension line in most cases. Only when the first extension line and the second extension line are completely parallel is it determined that the first extension line and the second extension line do not intersect. At this time, the current approximate line segment combination is skipped, and the next approximate line segment combination is traversed. Step 2, when it is determined that the first extension line and the second extension line intersect, it is further determined whether the target intersection point P of the first extension line and the second extension line is located within the effective area of ​​the current building facade image. The effective region is defined as the rectangular area enclosed by the image boundary of the current building facade image. That is, the effective region is the area with coordinates satisfying 0 ≤ x ≤ W and 0 ≤ y ≤ H, where W is the width of the building facade image (in pixels) and H is the height of the building facade image (in pixels). If the coordinates of the target intersection point P are ( , ) satisfies 0≤ ≤W and 0≤ If the value is less than or equal to H, then the target intersection point P is determined to be within the valid region; otherwise, the target intersection point P is determined to be outside the valid region, the current approximate line segment combination is skipped, and the next approximate line segment combination is traversed.

[0064] In the above embodiment, step three involves calculating the coordinates of the target intersection point P when it is determined that the target intersection point P is located within the effective area. Let the coordinates of the two endpoints of the approximately horizontal line segment of the target be (…). , )and( , The coordinates of the two endpoints of the approximately perpendicular line segment of the target are ( , )and( , Let the line containing the approximately horizontal segment of the target be denoted as... Let the line segment approximately perpendicular to the target be denoted as... The target intersection point P is calculated using the following formula:

[0065]

[0066] The intersection function finds two straight lines. and The intersection point. Specifically, the straight line containing the first extension. The equation is:

[0067]

[0068] The straight line containing the second extension line The equation is:

[0069]

[0070] Solving the equations of the two lines above simultaneously, we obtain the coordinates of the target intersection point P. , Step four: Determine the target intersection point P as the intersection point of the linear floor lines, and record the intersection point of the linear floor lines and its corresponding coordinates. , Add the points to the set of intersections of the floor lines. Repeat steps one through four of the above judgment and addition process until every approximate line segment combination in the set of approximate line segment combinations has been traversed. The final structured feature point set includes the set of intersections of the floor lines.

[0071] It should also be noted that in practical applications, if the target is approximately a horizontal line segment... approximately perpendicular line segment to the target They do not intersect (i.e., the target is approximately a horizontal line segment) approximately perpendicular line segment to the target If there are no common points within their respective endpoints, but the first and second extended lines intersect and the target intersection point is located within the valid area, then this intersection point can be retained as a hidden corner point of the building structure and added to the set of floor line intersection points. During the actual acquisition of building facade images, due to occlusion, shooting angle, or image cropping, some floor lines may not be completely detected. However, the intersection points of the extended lines corresponding to some floor lines still correspond to actual corner positions in the building structure. Retaining such hidden corner points can further enrich the feature information of the building facade and improve the completeness and accuracy of subsequent structural reconstruction of the building facade.

[0072] In an optional embodiment, a set of structured feature points is selected from a set of candidate corner points and a set of line segments based on the structural characteristics of the target building's facade. This includes: selecting a set of candidate pipe segments from the set of line segments that meet preset pipe segment conditions, where the preset pipe segment conditions are that the segment length is greater than a preset length threshold and the segment width is less than a preset width threshold; extracting the two endpoints of each candidate pipe segment in the set of candidate pipe segments to obtain a set of pipe endpoints; determining whether multiple candidate pipe segments in the set of candidate pipe segments intersect at the same location based on the set of pipe endpoints; when it is determined that multiple candidate pipe segments in the set of candidate pipe segments intersect at the same location, clustering the multiple endpoints of the multiple candidate pipe segments intersecting at the same location to obtain branch nodes; identifying the branch nodes and the endpoints in the set of pipe endpoints that did not participate in the clustering process as drainage pipe nodes, and adding the drainage pipe nodes to the set of drainage pipe nodes. The set of structured feature points includes the set of drainage pipe nodes.

[0073] Among them, the preset pipe segment conditions refer to the conditions used to screen candidate pipe segments, specifically, the segment length is greater than a preset length threshold and the segment width is less than a preset width threshold; the preset length threshold is the length threshold used to determine whether a segment has sufficient length to form a drainage pipe structure; the preset width threshold is the width threshold used to determine whether a segment has a slender feature to meet the width requirements of a drainage pipe structure; the candidate pipe segment set is the set of all segments that meet the preset pipe segment conditions selected from the segment set; the candidate pipe segment refers to each segment in the candidate pipe segment set; and the pipe endpoint set is the sum of the two endpoints of each candidate pipe segment extracted from the candidate pipe segment set. The set of all endpoints obtained; the same location refers to the location where the endpoints of multiple candidate pipeline segments are spatially clustered, that is, the distance between multiple endpoints is less than the preset clustering distance threshold; clustering processing refers to the process of spatially clustering multiple endpoints that intersect at the same location to determine branch nodes; branch node refers to the node representing the pipeline branch structure obtained after clustering multiple endpoints that intersect at the same location, and its coordinates are the mean of the coordinates of all endpoints within the cluster; drainage pipe node refers to all nodes representing the drainage pipe structure, including branch nodes and endpoints in the pipeline endpoint set that did not participate in the clustering processing; drainage pipe node set refers to the set of all drainage pipe nodes.

[0074] In the above embodiments, the drain pipe typically appears as a long, slender, vertical straight line segment, with nodes including endpoints, branch points, or turning points. (From the line segment set) The selection process involves filtering line segments that are longer and narrower as the candidate pipeline segment set. From the set of line segments The selection process involves filtering candidate pipeline segments that meet preset pipeline segment conditions. The preset conditions for pipeline segments are: the segment length is greater than a preset length threshold and the segment width is less than a preset width threshold. That is, for a set of segments... Each line segment in When line segment The length of the line segment is greater than the preset length threshold and the line segment When the width is less than the preset width threshold, the line segment will be... Add to candidate pipeline segment set In the middle. For the candidate pipeline segment set For each candidate pipeline segment, extract its two endpoints, and denote them as the first endpoint. Second endpoint :

[0075]

[0076] Set of candidate pipeline segments The first and second endpoints of all candidate pipe segments are summed to obtain the pipe endpoint set. According to the pipe endpoint set Determine the set of candidate pipeline segments Does the set of multiple candidate pipeline segments intersect at the same location? Specifically, iterate through the set of pipeline endpoints. Each endpoint in Statistical pipeline endpoint set Except for the endpoints Outside of, and endpoint The distance is less than the preset clustering distance threshold and endpoints The number of other endpoints belonging to different candidate pipeline segments is denoted as . .when When ≥2, it means except for the endpoints. Besides the candidate pipeline segment, there are at least two other candidate pipeline segments with endpoints and endpoints... If the endpoints of at least three candidate pipeline segments are spatially adjacent, meaning they cluster near the same location, it is determined that multiple candidate pipeline segments intersect at the same location, which constitutes a pipeline branch structure.

[0077] In the above embodiments, when determining the candidate pipeline segment set... When multiple candidate pipeline segments intersect at the same location, the pipeline endpoint set... All endpoints are clustered. Specifically, the DBSCAN clustering algorithm is used to cluster the pipeline endpoint set. Clustering is performed on all endpoints. The DBSCAN clustering algorithm has two input parameters: neighborhood radius parameter. (The value is the same as the preset clustering distance threshold) and the minimum number of points parameter (set to 3, indicating that at least 3 endpoints are needed to form a valid cluster). The clustering process of the DBSCAN clustering algorithm is as follows: For the pipeline endpoint set Each endpoint in , with end With the center, and Determine the endpoints for the radius The neighborhood range is defined, and the number of endpoints contained within that neighborhood range is counted (including endpoints). (Itself); if the number of endpoints is greater than or equal to the minimum number of points parameter, then the endpoints will be... Mark each endpoint as a core point. For each core point, all endpoints within its neighborhood are grouped into the same cluster, and this cluster is expanded in a chain-like manner using density reachability. That is, if an endpoint in the neighborhood is also a core point, then all endpoints in the neighborhood of that endpoint are also grouped into the same cluster, and so on, until the clusters can no longer be expanded. Endpoints not grouped into any cluster are marked as noise points. For each cluster output by the DBSCAN clustering algorithm, count how many different candidate pipeline segments each endpoint in the cluster comes from, and record this number as . .when When the value is ≥3, it indicates that at least 3 different candidate pipeline segments intersect at the same location. This cluster corresponds to a pipeline branching structure. The mean coordinates of all endpoints within the cluster are calculated, and this mean coordinate is used as the coordinates of the branch node to obtain the branch node. When the value is less than 3, the cluster does not form a branching structure, and the endpoints within the cluster do not generate branch nodes. All branch nodes obtained from the clustering process, endpoints marked as noise points by the DBSCAN clustering algorithm (i.e., isolated endpoints not assigned to any cluster), and... The endpoints within clusters with a value less than 3 are identified as drain pipe nodes. All drain pipe nodes are added to the drain pipe node set. In the end, the window corner point set. Floor line intersection set and drainage pipe node set The three sets of feature points are combined into a structured feature point set. .

[0078] In an optional embodiment, class weight enhancement is performed on the structured feature point set to obtain an enhanced feature point set, including: sequentially traversing each structured feature point in the structured feature point set, and performing the following operations on the currently traversed structured feature point: determining the target class to which the target structured feature point belongs, and determining the class weight coefficient of the target structured feature point according to the target class, wherein the target structured feature point is the currently traversed structured feature point; determining the corner response value and local feature point density of the target structured feature point; and determining the weight of the target structured feature point according to the class weight coefficient, corner response value, local feature point density, and a preset balance coefficient. Value; until each structured feature point has been traversed to obtain the weight value of each structured feature point; determine the Euclidean distance between any two structured feature points in the set of structured feature points; based on the Euclidean distance, if there exists a first structured feature point and a second structured feature point in the set of structured feature points whose Euclidean distance is less than a preset minimum distance threshold, compare the weight values ​​of the first structured feature point and the second structured feature point to obtain the weight comparison result; based on the weight comparison result, remove the first or second structured feature point that does not meet the preset weight retention condition from the set of structured feature points to obtain the enhanced feature point set.

[0079] Here, the target structured feature point refers to the structured feature point currently traversed; the target category refers to the category to which the target structured feature point belongs, which includes three categories: window corner points, floor line intersections, and drainage pipe nodes; the category weight coefficient refers to the weight coefficient determined according to the target category to which the target structured feature point belongs, with different category weight coefficients corresponding to different target categories; the corner response value refers to the measure of the salience of the target structured feature point as a corner point, calculated using the Harris corner response formula; and the local feature point density refers to the number of other structured feature points contained within a circular neighborhood centered on the target structured feature point and with a preset density statistical radius as the radius.

[0080] The preset balance coefficients include a first preset balance coefficient and a second preset balance coefficient. The first preset balance coefficient controls the contribution ratio of corner response values ​​to the weight values, and the second preset balance coefficient controls the contribution ratio of local feature point density to the weight values. The weight value refers to the weight of the target structured feature point calculated comprehensively based on the category weight coefficient, corner response value, local feature point density, and preset balance coefficients. The Euclidean distance refers to the Euclidean distance between any two structured feature points in the structured feature point set in the image coordinate space. The preset minimum distance threshold is a distance threshold used to determine whether two structured feature points are too close, usually taken as 5 to 15 pixels. The first structured feature point and the second structured feature point refer to any pair of structured feature points in the structured feature point set whose Euclidean distance is less than the preset minimum distance threshold. The weight comparison result refers to the relationship between the weight values ​​of the first structured feature point and the weight values ​​of the second structured feature point. The preset weight retention condition refers to the condition for retaining the structured feature point with the larger weight value among two structured feature points whose Euclidean distance is less than the preset minimum distance threshold.

[0081] In the above embodiments, the three types of building structural feature points (window corner points, floor line intersections, and drainage pipe nodes) that have already been selected are further enhanced. Weights are assigned according to the respective categories of the three types of building structural feature points, and then... Perform nonmaximum suppression based on weights and categories. The purpose of this step is to highlight these key structural points, allowing them to play a dominant role in the matching process. Then, iterate through the set of structured feature points sequentially. For each structured feature point in the dataset, assuming the target structured feature point currently being traversed is... And perform the following operations on the currently traversed structured feature points: Step 1, the target categories include three types: window corner points, floor line intersections, and drainage pipe nodes. The category weight coefficients corresponding to different target categories... The values ​​are as follows: when the target structured feature point When it belongs to a window corner point, the category weight coefficient Take 1.5; when the target structured feature points When it is the intersection of floor lines, the category weight coefficient Set to 1.0; when the target structured feature points When it belongs to a drainage pipe node, the category weight coefficient Set to 1.0. Category weight coefficient for window corner points. Category weighting coefficients for intersections of floor lines and drainage pipe nodes This is because window corner points have stronger geometric saliency and a higher repeatability detection rate in building facade images, providing more reliable constraints in image matching. Step two involves using target structured feature points. Using the coordinates of the center position as the reference point, calculate the autocorrelation matrix M within the neighborhood of this center position (the neighborhood size is typically 5×5 or 7×7 pixels, etc.). Then, calculate the Harris corner response value according to the following formula. :

[0082]

[0083] in, Let M be the determinant of the autocorrelation matrix. It is the trace of the autocorrelation matrix M (i.e., the sum of the diagonal elements of the matrix). This is an empirical constant (usually taken as 0.04~0.06). Corner response value. Reflecting the structured feature points of the target Corner response value is a measure of the salience of a corner. The larger the value, the more structured feature points the target represents. The more pronounced the corner characteristics, the better.

[0084] In the above embodiment, step three, local feature point density Defined as target structured feature points Centered on a circle with a preset density statistical radius The set of structured feature points contained within a circular neighborhood of radius . The number of other structured feature points. Preset density statistical radius. The value of is determined based on the resolution of the building facade image, typically ranging from 20 to 50 pixels. The formula for calculating the local feature point density is:

[0085]

[0086] in, The cardinality of a set is the number of elements in the set. Represents a structured feature point set Other structured feature points that meet the following two conditions The set: Condition 1 is Not equal to the target structured feature points (i.e., exclude) (Itself); condition 2 is and The Euclidean distance between them is less than the preset density statistical radius. . Representing the target structured feature points Other structured feature points The Euclidean distance between them is calculated using the following formula:

[0087]

[0088] in, Target structured feature points x-coordinate Target structured feature points The ordinate, For other structured feature points horizontal, For other structured feature points The ordinate. Local feature point density. Used to measure target structured feature points The density of surrounding feature points, the density of local feature points The larger the value, the more structured feature points the target represents. The denser the surrounding feature points, the better. Step 4: Weight values. The calculation formula is:

[0089]

[0090] in, The first preset balance coefficient is used to control the corner response value. In weight value The proportion of contribution in The value range is 0.5 to 1.0, and it is usually taken as 0.7; The second preset balance coefficient is used to control the density of local feature points. In weight value The proportion of contribution in The value ranges from 0.3 to 0.8, and is typically 0.5. The above weight values... In the calculation formula, The term indicates that structured feature points with higher corner response values ​​receive higher weights; The term indicates that structured feature points with lower local feature point density (i.e., sparser surrounding feature points) receive higher weights, thereby suppressing feature points in overly dense areas and encouraging the retention of feature points in sparse areas. As an overall scaling factor, structured feature points of different categories receive weight amplification corresponding to their category importance. Repeat steps one through four above until the entire set of structured feature points has been traversed. For each structured feature point, obtain the weight value for each structured feature point. .

[0091] In the above embodiments, structured feature points that are too close to each other are suppressed. The specific process is as follows: Calculate the set of structured feature points. The Euclidean distance between any two structured feature points in the set. The first structured feature point exists in it. With the second structured feature point Euclidean distance between Less than the preset minimum distance threshold At that time, the first structured feature point weight value With the second structured feature point weight value Perform weighted comparison. Preset minimum distance threshold. The value ranges from 5 to 15 pixels, but is typically set to 10 pixels. The specific operation of the weighted comparison is as follows: compare the first structured feature points. weight value With the second structured feature point weight value The relative sizes of the distances are used to obtain the weighted comparison results. The preset weight retention condition is: when the Euclidean distance is less than a preset minimum distance threshold... Of the two structured feature points, the structured feature point with the larger weight value satisfies the preset weight retention condition, while the structured feature point with the smaller weight value does not satisfy the preset weight retention condition. That is, when At that time, retain the first structured feature point. The second structured feature point From the set of structured feature points Remove from the middle; when At that time, the second structured feature points are preserved. The first structured feature point From the set of structured feature points Remove from the middle. When = At that time, the first structured feature points are randomly retained. Or the second structured feature point One of them will take another from the set of structured feature points. Remove from the middle. For the structured feature point set... All values ​​whose Euclidean distance is less than a preset minimum distance threshold All structured feature point pairs undergo the aforementioned weight comparison and elimination operations. After this filtering and enhancement, an optimized set of structured feature points (i.e., the enhanced feature point set) is obtained, denoted as […]. Strengthen feature point set Each feature point in Includes the following information:

[0092]

[0093] in, For feature points The coordinates; For feature points The scale; For feature points Descriptors; For feature points Category labels ( (These correspond to the window corner, the intersection of the floor lines, and the drain pipe node, respectively). For feature points The weight values ​​are then used to enhance the feature point set for subsequent image matching steps.

[0094] In an optional embodiment, the first adjacent image and the second adjacent image are synthesized according to the matching point pair set to obtain the target panoramic image, including: performing the following iterative operation on the matching point pair set according to a preset number of iterations: randomly selecting a preset number of non-collinear matching point pairs from the matching point pair set and determining the candidate homography matrix of the non-collinear matching point pairs; determining the projection error of other matching point pairs in the matching point pair set according to the candidate homography matrix, wherein the other matching point pairs are the remaining matching point pairs in the matching point pair set excluding the non-collinear matching point pairs; and converting the projection error of other matching point pairs with a projection error less than a preset error threshold into a single image. For each inlier, an inlier is identified and its number is determined. This iterative process continues until a preset number of iterations are completed, yielding multiple candidate homography matrices and the number of inliers corresponding to each candidate homography matrix. Based on the number of inliers, the candidate homography matrices that satisfy the preset inlier number condition are identified as the target homography matrix. All inliers corresponding to the target homography matrix are identified as the target inlier set. The target homography matrix is ​​re-estimated and optimized based on the target inlier set to obtain an optimized homography matrix. The first adjacent image and the second adjacent image are stitched together based on the optimized homography matrix to obtain the target panoramic image.

[0095] Here, the preset iteration count refers to the total number of iterations of the RANSAC algorithm for random sampling and model validation, typically between 500 and 1000; the preset quantity refers to the number of non-collinear matching point pairs randomly selected from the matching point pair set in each iteration, with a preset quantity of 4 pairs; a non-collinear matching point pair refers to 4 pairs of matching points randomly selected from the matching point pair set, requiring that the positions of these 4 matching points in the first adjacent image are not collinear; the candidate homography matrix refers to the 3×3 projection transformation matrix calculated using the non-collinear matching point pairs randomly selected in each iteration; other matching point pairs refer to the remaining matching point pairs in the matching point pair set excluding the non-collinear matching point pairs selected in the current iteration; and the projection error refers to the difference between the projected position and the corresponding matching point in the second adjacent image after projecting the matching point in the first adjacent image onto the second adjacent image using the candidate homography matrix. Euclidean distance; the preset error threshold is the projection error threshold used to determine whether a matching point pair is an inlier, usually set to 3 to 5 pixels; an inlier is a matching point pair whose projection error is less than the preset error threshold; the number of inliers is the number of matching point pairs determined to be inliers under the current candidate homography matrix; the preset inlier number condition is the condition for selecting the candidate homography matrix with the most inliers among all candidate homography matrices; the target homography matrix is ​​the candidate homography matrix that satisfies the preset inlier number condition; the target inlier set is the set of all inliers corresponding to the target homography matrix; re-estimation optimization is the process of re-estimating the homography matrix using all inliers in the target inlier set, employing the least squares method or nonlinear optimization method to minimize the sum of squared projection errors of all inliers; the optimized homography matrix is ​​the homography matrix with higher accuracy obtained after re-estimation optimization.

[0096] In the above embodiments, the RANSAC algorithm is used to start from the initial set of matching point pairs. The algorithm iteratively selects the best matching point pairs, eliminates mismatches, estimates the accurate homography matrix, and then completes image transformation and seamless fusion to generate a high-quality stitched image. The RANSAC algorithm uses iterative random sampling and model validation to select the best matching point pairs from an initial set containing a large number of mismatches. The optimal set of interior points (i.e., the set of best-matching point pairs) is selected from the data, and the most accurate homography matrix is ​​estimated. In each iteration, perform the following steps: Step 1, from Four pairs of non-collinear matching points are randomly selected from the data. , Using these 4 pairs of matching points, the candidate homography matrix is ​​calculated. The homography matrix describes the first neighboring image. Second adjacent image Projection transformation relationship between them:

[0097]

[0098] in, The first neighboring image Homogeneous coordinates of the matching points in the reference image. For the second neighboring image The homogeneous coordinates of the matching points in the target image (i.e., the target image). The above projection transformation relationship indicates that the second adjacent image... Points in Through homography matrix Projection transformation to the first adjacent image The corresponding position in Specifically, based on the projection transformation relationship, for each pair of matching points ( , ), and the second adjacent image middle matching point coordinates ( , Substituting the projection transformation formula and performing homogeneous coordinate normalization, we obtain the following two equations:

[0099]

[0100]

[0101] in,( , () is the first neighboring image middle matching point The known coordinates, ( , () is the second neighboring image middle matching point The known coordinates, to Candidate homography matrix The unknown parameters. Each matching point pair provides the above two equations, and four non-collinear matching point pairs provide a total of eight equations. Candidate homography matrix. There are a total of 9 unknown parameters ( to ),make After normalization, the remaining eight unknown parameters are determined by exactly eight equations. Solving these linear equations yields the candidate homography matrix. .

[0102] In the above embodiment, step two involves using the current candidate homography matrix to... The projection error is calculated for all matching points except the sampling points. The formula for calculating the projection error is:

[0103]

[0104] Other matching point pairs ( , Substituting into the projection error calculation formula, we get:

[0105]

[0106] in,( , () is the first neighboring image middle matching point coordinates, ( , () is the second neighboring image middle matching point coordinates to These are the elements of the current candidate homography matrix. Projection error. This indicates that the first adjacent image Matching points in Project the current candidate homography matrix onto the second neighboring image. Then, the projection position is the same as the second adjacent image. Actual matching point The Euclidean distance between them. Step 3: Set the error threshold. (Usually 3-5 pixels). If Then the matching point is determined to be the current candidate homography matrix. If a point is an interior point, it is an exterior point; otherwise, it is an exterior point. Calculate the current candidate homography matrix. Number of interior points below Repeat steps one through three of the above process for random sampling, model calculation, and interior point statistics, performing a total of K iterations (K is the preset number of iterations, typically 500-1000). Record the number of interior points in each iteration. The process continues until K iterations are completed, yielding K candidate homography matrices and the number of interior points corresponding to each candidate homography matrix.

[0107] In the above embodiment, after the iteration is completed, the candidate homography matrix with the largest number of interior points is selected from the K candidate homography matrices as the target homography matrix. :

[0108]

[0109] Among them, in all candidate homography matrices In the middle, select the number of interior points. The largest candidate homography matrix is ​​used as the target homography matrix. The preset condition for the number of interior points is that the number of interior points is maximized among all candidate homography matrices. The target homography matrix... All corresponding interior points are determined as the target interior point set I. This is achieved using the target homography matrix. For all interior points in the corresponding target interior point set I, the homography matrix is ​​re-estimated to further improve accuracy. Typically, the least squares method or a nonlinear optimization method (such as the Levenberg-Marquardt algorithm) is used to minimize the sum of squared projection errors of all interior points in the target interior point set I.

[0110]

[0111] Where I is the set of points inside the target; , () is the first neighboring image center point The coordinates (as input to the candidate homography matrix H); , () is the second neighboring image Corresponding interior point The actual coordinates (as a comparison benchmark); the 2×1 vector on the left is the second adjacent image. center point The actual coordinates; the 2×1 vector on the right is the first adjacent image. center point Projected onto the second adjacent image via the homography matrix. The projected coordinates are shown below; H is the variable to be optimized. to Let H be the elements. The optimization objective is to find the homography matrix that minimizes the sum of squared projection errors of all interior points in the target interior point set I. The optimized homography matrix is ​​finally obtained. (i.e., the precise homography matrix), while the target interior point set I is the selected best matching point pair.

[0112] In an optional embodiment, stitching a first adjacent image and a second adjacent image together according to an optimized homography matrix to obtain a target panoramic image includes: performing a perspective transformation on the second adjacent image according to the optimized homography matrix to obtain a transformed image; creating a blank canvas and placing the first adjacent image in the blank canvas according to the original coordinates to obtain a first canvas image; determining the transformation coordinates of the transformed image according to the optimized homography matrix and placing the transformed image in the blank canvas according to the transformation coordinates to obtain a second canvas image; determining the overlapping area between the first canvas image and the second canvas image, the first non-overlapping area of ​​the first canvas image and the second non-overlapping area of ​​the second canvas image according to the positional relationship between the first canvas image and the second canvas image in the blank canvas; constructing a first Gaussian pyramid for the first adjacent image, and constructing a second Gaussian pyramid for the transformed image; and applying the first Gaussian pyramid... The first adjacent layer difference processing is performed on the pyramid to generate a first Laplacian pyramid, and the second adjacent layer difference processing is performed on the second Gaussian pyramid to generate a second Laplacian pyramid. A bottom weight map is constructed based on the overlapping region, the first non-overlapping region, and the second non-overlapping region. In the bottom weight map, the pixel weight value located in the first non-overlapping region is a first preset weight value, the pixel weight value located in the second non-overlapping region is a second preset weight value, and the pixel weight value located in the overlapping region changes linearly from the first preset weight value to the second preset weight value along the direction from the first boundary between the overlapping region and the first non-overlapping region to the second boundary between the overlapping region and the second non-overlapping region. The bottom weight map is decomposed into a weight pyramid, and the first Laplacian pyramid and the second Laplacian pyramid are fused into a multi-resolution image based on the weight pyramid to obtain the target panoramic image.

[0113] In this context, perspective transformation refers to the process of transforming the projected coordinates of each pixel in the second adjacent image using an optimized homography matrix; the transformed image refers to the image obtained after performing perspective transformation on the second adjacent image; the blank canvas refers to a blank image created in a sufficiently large size to simultaneously contain the entire contents of the first adjacent image and the transformed image; the original coordinates refer to the original position coordinates of the pixels in the first adjacent image; the first canvas image refers to the canvas image formed by placing the first adjacent image in the blank canvas according to its original coordinates; the transformed coordinates refer to the position coordinates of each pixel in the transformed image in the blank canvas, calculated using the optimized homography matrix; the second canvas image refers to the canvas image formed by placing the transformed image in the blank canvas according to its transformed coordinates; and the overlapping region refers to the area where both the first canvas image and the second canvas image simultaneously contain pixels in the blank canvas. The regions are defined as follows: the first non-overlapping region refers to the region containing only pixels of the first adjacent image and not pixels of the transformed image; the second non-overlapping region refers to the region containing only pixels of the transformed image and not pixels of the first adjacent image; the first Gaussian pyramid refers to the multi-scale Gaussian pyramid constructed for the first adjacent image, obtained through layer-by-layer Gaussian filtering and downsampling; the second Gaussian pyramid refers to the multi-scale Gaussian pyramid constructed for the transformed image, constructed in the same way as the first Gaussian pyramid; the first adjacent layer difference processing refers to the processing of performing difference operations on the adjacent layers of the first Gaussian pyramid, that is, subtracting the upsampled and magnified image from the current layer image; the first Laplacian pyramid refers to the Laplacian pyramid generated after performing the first adjacent layer difference processing on the first Gaussian pyramid, which records the high-frequency detail information of the first adjacent image at each scale.

[0114] Among them, the second adjacent layer difference processing refers to the processing of differential operation on the adjacent layers of the second Gaussian pyramid; the second Laplacian pyramid refers to the Laplacian pyramid generated after the second adjacent layer difference processing of the second Gaussian pyramid, which records the high-frequency detail information of the transformed image at various scales. The bottom-level weight map refers to an image constructed based on overlapping regions, a first non-overlapping region, and a second non-overlapping region, defining the fusion weight at each pixel location. Pixel weights within the first non-overlapping region are defined by a first preset weight value of 1, while those within the second non-overlapping region are defined by a second preset weight value of 0. Pixel weights within the overlapping region linearly change from the first preset weight value of 1 to the second preset weight value of 0. A first preset weight value of 1 indicates that the pixel values ​​of the first adjacent image are used entirely; a second preset weight value of 0 indicates that the pixel values ​​of the transformed image are used entirely. The first boundary refers to the dividing line between the overlapping region and the first non-overlapping region; the second boundary refers to the dividing line between the overlapping region and the second non-overlapping region. Multi-scale decomposition processing refers to the process of performing layer-by-layer Gaussian filtering and downsampling on the bottom-level weight map to construct a weight pyramid. The weight pyramid refers to the set of multi-layer weighted images obtained after multi-scale decomposition processing of the bottom-level weight map. Multi-resolution fusion processing refers to the process of weighted fusion of the first and second Laplacian pyramids at each layer based on the weight pyramid, and then upsampling and accumulating from the highest layer to reconstruct the final fused image.

[0115] In the above embodiments, the first adjacent image Using the reference image as a baseline, the estimated optimized homography matrix is ​​utilized. For the second adjacent image Perform perspective transformation on the target image to obtain the transformed image. For the second adjacent image Each pixel position in ( , Using the optimized homography matrix Calculate the position of each pixel ( , ) in the first adjacent image Transformed coordinates in coordinate system , ):

[0116]

[0117] in,( , () is the second neighboring image The original coordinates of the middle pixel. For the optimized homography matrix Transformed homogeneous coordinates ( , The transformed coordinates are obtained after normalizing the aligned secondary coordinates, i.e., the second adjacent image. Pixels in , ) in the first adjacent image The corresponding position in the coordinate system. Due to coordinate transformation ( , The value may be a non-integer value, requiring bilinear interpolation to determine the pixel value at the transformed coordinates, thus obtaining the transformed image. Create a blank canvas large enough to simultaneously hold the first adjacent image. and transform images The entire content. The first adjacent image. Place it on the blank canvas according to the original coordinates to obtain the first canvas image. The first canvas image is placed on the blank canvas according to the transformed coordinates, resulting in the second canvas image. Based on the positional relationship between the first and second canvas images on the blank canvas, the following three regions are determined: overlapping region. The area where the first canvas image and the second canvas image overlap in the blank canvas, that is, the area that simultaneously contains the first adjacent image. Pixels and transformed images The region of pixels. The first non-overlapping region. Only the first adjacent image is included Pixels, excluding transformed images The region of pixels. The second non-overlapping region. Only includes transformed images Pixels that do not contain the first adjacent image The region of pixels.

[0118] In the above embodiment, it is the first adjacent image. Constructing the First Gaussian Pyramid , for transforming images Constructing the Second Gaussian Pyramid ,in Number the pyramid levels ( The value is determined based on the image size, and is usually taken as... ,in and (These represent the width and height of the image, respectively). First Gaussian Pyramid The construction process is as follows: hour, The first neighboring image The original image; for , to the upper layer After performing Gaussian filtering (using a 5×5 Gaussian convolution kernel) and then downsampling to 2×2 (i.e., reducing the image width and height to half of their original values), the result is... The Second Pyramid of Gauss The construction process of the first Gaussian pyramid The same, except the input image is replaced with the transformed image. Regarding the first Gaussian pyramid Perform the first adjacent layer difference processing to generate the first Laplace pyramid. The formula for differential processing of the first adjacent layer is:

[0119]

[0120] in, The first Gaussian pyramid Layer image, The first Gaussian pyramid Layer image, This indicates the first Gaussian pyramid. Layer Image Perform upsampling operation, Enlarge to the number Layer Image The size. For the first Laplace pyramid Layer image, representing the first Gaussian pyramid. Layer Image With the Layer Image The difference after upsampling and amplification. (Regarding the second Gaussian pyramid) Perform a second adjacent layer difference processing to generate the second Laplace pyramid. The formula for differential processing of the second adjacent layer is:

[0121]

[0122] in, The second Gaussian pyramid Layer image, The second Gaussian pyramid Layer image, This indicates the first... Layer Image Perform upsampling operation, Enlarge to the number Layer Image The size. For the second Laplace pyramid Layer image, representing the second Gaussian pyramid. Layer Image With the Layer Image The difference after upsampling and amplification. The operation consists of two steps: upsampling and Gaussian convolution. Specifically, the smaller Gaussian pyramid image is first processed... Insert one row / column of zero-value pixels in both the row and column directions (i.e., upsampling), then perform a 5×5 Gaussian convolution operation (i.e., Gaussian smoothing) on ​​the image after inserting the zero values, thereby... Zoom in to the current layer The size.

[0123] In the above embodiments, The specific calculation formula for the operation is as follows:

[0124]

[0125] Among them, subscript Indicates image number ( Corresponding to the first adjacent image, (corresponding transformed image) The pixel position in the upsampled image; Position in a 5×5 Gaussian convolution kernel The weighting coefficient at the location, and The range of values ​​for is [−2, 2]; when or When it is a non-integer, The value is 0. The coefficient 4 is the normalization scaling factor, used to compensate for zero-value pixels inserted during upsampling. 5×5 Gaussian convolution kernel. The specific values ​​are:

[0126]

[0127] Each level of the Pyramid of Laplace Recorded the image of the current layer of the Gaussian pyramid. Image of the Gaussian pyramid above The difference after upsampling and amplification represents the high-frequency detail information of the current layer. The highest layer of the Laplace pyramid. = That is, the image of the highest level of the Gaussian pyramid is directly taken as the highest level of the Laplace pyramid.

[0128] In the above embodiments, based on the overlapping region First non-overlapping region Second non-overlapping region Constructing the underlying weight graph Bottom weight graph Defines the position of each pixel in the blank canvas. The first adjacent image Fusion weight values:

[0129]

[0130] Where, when pixel position Located in the first non-overlapping region When the weight value is 1 (i.e., the first preset weight value), it means that the pixel position fully utilizes the first neighboring image. The pixel value. When the pixel position Located in the second non-overlapping region When the weight value is 0 (i.e., the second preset weight value), it means that the pixel position is completely transformed using the image. The pixel value. When the pixel position Located in the overlapping region During this period, the weight value linearly changes from 1 to 0 along the direction from the first boundary between the overlapping region and the first non-overlapping region to the second boundary between the overlapping region and the second non-overlapping region. Wherein, The x-coordinate of the left boundary of the overlapping region (i.e., the x-coordinate of the first boundary). This represents the x-coordinate of the right boundary of the overlapping region (i.e., the x-coordinate of the second boundary). For the underlying weight map... Gaussian filtering and 2×2 downsampling are performed to obtain the weight pyramids for each layer. The weighted pyramid is constructed in the same way as the Gaussian pyramid. Based on the weighted pyramid... The First Laplace Pyramid Second Laplace Pyramid By performing a weighted fusion layer by layer, we obtain the fused Laplace pyramid. :

[0131]

[0132] in, For the first Pixel position in layer weight pyramid The weight value at that location, For the first Laplace pyramid Pixel position in layer Pixel value at that location, For the second Laplace pyramid Pixel position in layer The pixel values ​​at each location. Starting from the highest layer of the fused Laplacian pyramid, upsampling and accumulating layer by layer are performed to reconstruct the final fused image:

[0133]

[0134]

[0135] in, To integrate the images of the highest level of the Laplace pyramid, To merge the image from the previous layer The result is upsampled to the current layer size. This is the fused image for the current layer. Final. This results in a seamlessly stitched panoramic image. The generated seamlessly stitched image... The effective information of both images was fully preserved, with no obvious stitching marks, and can be directly used for subsequent building facade damage analysis.

[0136] Through the embodiments of this application, the building facade image stitching system sequentially performs multi-scale feature extraction, three types of geometric feature point selection based on the structural characteristics of the building facade (window corner points, floor line intersections, and drainage pipe nodes), category weight enhancement, feature matching based on Euclidean distance nearest neighbor search and Lowe's ratio test, selection of best matching point pairs based on RANSAC algorithm and optimization of homography matrix estimation, and seamless stitching based on multi-resolution fusion to generate a target panoramic image with high structural integrity.

[0137] The following describes the building facade image stitching system in the embodiments of this invention from the perspective of hardware processing. (See attached document.) Figure 2 , Figure 2 This is a schematic diagram of the physical device structure of a building facade image stitching system in the embodiments of this application.

[0138] It should be noted that, Figure 2 The structure of the building facade image stitching system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0139] like Figure 2 As shown, the building facade image stitching system includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in Read-Only Memory (ROM) 202 or a program loaded from storage section 208 into Random Access Memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores...

[0140] The system contains various programs and data required for operation. CPU 201, ROM 202, and RAM 203 are interconnected via bus 204. Input / output (I / O) interface 205 is also connected to bus 204. The following components are connected to I / O interface 205: input section 206, including audio input devices, push-button switches, etc.; output section 207, including a liquid crystal display (LCD), audio output devices, indicator lights, etc.; storage section 208, including a hard disk, etc.; and communication section 209, including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0141] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the various functions defined in the present invention.

[0142] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than that shown in the drawings.

[0144] Specifically, the building facade image stitching system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the building facade image stitching method based on UAV single-view continuous images provided in the above embodiment.

[0145] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the building facade image stitching system described in the above embodiments; or it may exist independently and not assembled into the building facade image stitching system. The storage medium carries one or more computer programs, which, when executed by a processor of the building facade image stitching system, cause the building facade image stitching system to implement the building facade image stitching method based on UAV single-view continuous imagery provided in the above embodiments.

[0146] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A building facade image stitching method based on a single view continuous image of a UAV, characterized in that, include: Multiple exterior facade images of the target building are continuously collected from a single viewpoint by a drone, and features are extracted from each of the multiple exterior facade images to obtain an initial feature point set. Corner detection is performed on the initial feature point set to obtain a candidate corner point set. Line segment detection is performed on each building facade image to obtain a line segment set. A structured feature point set is then selected from the candidate corner point set and the line segment set based on the building facade structural characteristics of the target building. The structured feature point set is enhanced by class weighting to obtain an enhanced feature point set; From the plurality of building facade images, a first adjacent image and a second adjacent image are determined, and from the enhanced feature point set, a first feature point set of the first adjacent image and a second feature point set of the second adjacent image are determined; The first feature point set and the second feature point set are matched to obtain a set of matching point pairs. The first adjacent image and the second adjacent image are synthesized based on the set of matching point pairs to obtain the target panoramic image.

2. The method of claim 1, wherein, The step of selecting the set of structured feature points from the candidate corner point set and the line segment set based on the architectural facade structural characteristics of the target building includes: Iterate through each candidate corner in the candidate corner set, and perform the following operation on the currently traversed candidate corner: From the set of line segments, select horizontal line segments whose first angle with the horizontal direction is less than a first preset angle threshold and whose first shortest distance to the target corner point is less than a preset distance threshold. The target corner point is the candidate corner point currently traversed. The line segments are selected from the set of line segments that have a second included angle with the vertical direction that is less than a second preset angle threshold and a second shortest distance from the target corner point that is less than the preset distance threshold, wherein the first preset angle threshold is less than the second preset angle threshold; When both the horizontal line segment and the vertical line segment are selected simultaneously, the target corner point is determined to be the corner point of the planar window formed by the horizontal line segment and the vertical line segment. The planar window corner points are added to the window corner point set until each candidate corner point has been traversed. The structured feature point set includes the window corner point set.

3. The method according to claim 1, characterized in that, The step of selecting the set of structured feature points from the candidate corner point set and the line segment set based on the architectural facade structural characteristics of the target building includes: Extract an approximate set of horizontal line segments and an approximate set of vertical line segments from the set of line segments; By combining the approximate horizontal line segment set with the approximate vertical line segment set in pairs, multiple approximate line segment combination sets are obtained; Iterate through each of the multiple sets of approximate line segment combinations in turn, and perform the following operation on the currently traversed approximate line segment combination: Determine whether the first extension of the target approximate horizontal line segment intersects with the second extension of the target approximate vertical line segment. The currently traversed approximate line segment combination includes the target approximate horizontal line segment and the target approximate vertical line segment. When determining that the first extension line intersects with the second extension line, determine whether the target intersection point of the first extension line and the second extension line is located within the effective area of ​​the current building facade image; When it is determined that the target intersection point is located within the effective area, the coordinates of the target intersection point are determined; The target intersection point is determined as the linear floor line intersection point, and the linear floor line intersection point is added to the floor line intersection point set until each approximate line segment combination is traversed. The structured feature point set includes the floor line intersection point set.

4. The method according to claim 1, characterized in that, The step of selecting the set of structured feature points from the candidate corner point set and the line segment set based on the architectural facade structural characteristics of the target building includes: From the set of line segments, a set of candidate pipeline line segments that meet the preset pipeline line segment conditions are selected, wherein the preset pipeline line segment conditions are that the line segment length is greater than a preset length threshold and the line segment width is less than a preset width threshold. Extract the two endpoints of each candidate pipe segment from the candidate pipe segment set to obtain the pipe endpoint set; Based on the set of pipeline endpoints, determine whether there are multiple candidate pipeline segments in the set of candidate pipeline segments that intersect at the same location; When it is determined that multiple candidate pipeline segments intersect at the same location in the candidate pipeline segment set, the multiple endpoints of the multiple candidate pipeline segments intersecting at the same location are clustered to obtain branch nodes; The branch nodes and the endpoints in the pipe endpoint set that did not participate in the clustering process are identified as drainage pipe nodes, and the drainage pipe nodes are added to the drainage pipe node set. The structured feature point set includes the drainage pipe node set.

5. The method according to claim 1, characterized in that, The process of strengthening the structured feature point set by category weights to obtain a strengthened feature point set includes: Iterate through each structured feature point in the set of structured feature points in sequence, and perform the following operation on the currently traversed structured feature point: Determine the target category to which the target structured feature point belongs, and determine the category weight coefficient of the target structured feature point based on the target category, wherein the target structured feature point is the structured feature point currently traversed; Determine the corner response values ​​and local feature point density of the target structured feature points; The weight value of the target structured feature point is determined based on the category weight coefficient, the corner response value, the local feature point density, and the preset balance coefficient. This process continues until all structured feature points have been traversed to obtain the weight value of each structured feature point. Determine the Euclidean distance between any two structured feature points in the set of structured feature points; If, in the set of structured feature points determined by the Euclidean distance, the Euclidean distance between the first structured feature point and the second structured feature point is less than a preset minimum distance threshold, the weight values ​​of the first structured feature point and the second structured feature point are compared to obtain a weight comparison result. Based on the weight comparison results, the first or second structured feature points that do not meet the preset weight retention conditions are removed from the set of structured feature points to obtain the enhanced feature point set.

6. The method according to claim 2, characterized in that, The step of combining the first adjacent image and the second adjacent image according to the set of matching point pairs to obtain the target panoramic image includes: Perform the following iterative operation on the set of matching point pairs according to a preset number of iterations: A predetermined number of non-collinear matching point pairs are randomly selected from the set of matching point pairs, and a candidate homography matrix of the non-collinear matching point pairs is determined. The projection error of other matching point pairs in the matching point pair set is determined based on the candidate homography matrix, wherein the other matching point pairs are the remaining matching point pairs in the matching point pair set excluding the non-collinear matching point pairs; Other matching point pairs whose projection error is less than a preset error threshold are identified as interior points, and the number of interior points is determined. The iteration operation continues until the preset number of iterations is completed, in order to obtain multiple candidate homography matrices and the number of interior points corresponding to each candidate homography matrix; Based on the number of interior points, the candidate homography matrix that satisfies the preset number of interior points among the plurality of candidate homography matrices is determined as the target homography matrix; The set of all interior points corresponding to the target homography matrix is ​​determined as the target interior point set; The target homography matrix is ​​re-estimated and optimized based on the target interior point set to obtain the optimized homography matrix; The first adjacent image and the second adjacent image are stitched together according to the optimized homography matrix to obtain the target panoramic image.

7. The method according to claim 6, characterized in that, The step of stitching the first neighboring image and the second neighboring image together according to the optimized homography matrix to obtain the target panoramic image includes: The second adjacent image is subjected to perspective transformation based on the optimized homography matrix to obtain the transformed image; Create a blank canvas and place the first adjacent image in the blank canvas according to the original coordinates to obtain the first canvas image; The transformation coordinates of the transformed image are determined according to the optimized homography matrix, and the transformed image is placed in the blank canvas according to the transformation coordinates to obtain the second canvas image; Based on the positional relationship between the first canvas image and the second canvas image in the blank canvas, the overlapping area between the first canvas image and the second canvas image, the first non-overlapping area of ​​the first canvas image, and the second non-overlapping area of ​​the second canvas image are determined. A first Gaussian pyramid is constructed for the first adjacent image, and a second Gaussian pyramid is constructed for the transformed image; The first Gaussian pyramid is subjected to a first adjacent layer difference process to generate a first Laplace pyramid, and the second Gaussian pyramid is subjected to a second adjacent layer difference process to generate a second Laplace pyramid. A bottom-level weight map is constructed based on the overlapping region, the first non-overlapping region, and the second non-overlapping region. The underlying weight map is decomposed into a weight pyramid at multiple scales to obtain a weight pyramid. The first Laplacian pyramid and the second Laplacian pyramid are then fused at multiple resolutions based on the weight pyramid to obtain the target panoramic image.

8. A building facade image stitching system, characterized in that, The building facade image stitching system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the building facade image stitching system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising program instructions, characterized in that, When the program instructions are run on the building facade image stitching system, the building facade image stitching system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the building facade image stitching system, the building facade image stitching system performs the method as described in any one of claims 1-7.

Images