A multi-unmanned aerial vehicle vision cooperative building scene identification method

Abstract

The application relates to the technical field of image processing, and discloses a building scene recognition method based on multi-unmanned aerial vehicle (UAV) vision cooperation, which comprises the following steps: acquiring building multi-view images and accurate six-degree-of-freedom poses through a time-space synchronous multi-UAV cluster; calculating a fundamental matrix between image pairs based on the poses, and establishing an epipolar geometric constraint; extracting candidate feature points and constructing an initial matching graph; performing joint verification of geometric consistency and photometric consistency - the former verifies spatial consistency by using an epipolar constraint and triangulation, and the latter analyzes the brightness variation law with the view angle based on a Fresnel reflection model to distinguish diffuse reflection from specular reflection; according to the above, the feature points are divided into a structural feature point set and a reflection feature point set, and only the former is used for motion recovery structure and dense reconstruction. Through a physical and geometric double verification mechanism, the application effectively eliminates reflection interference, and significantly improves the geometric precision and integrity of a building three-dimensional model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for building scene recognition using multi-UAV visual collaboration. Background Technology

[0002] With the widespread application of drone swarm technology in smart cities, architectural surveying, and disaster emergency response, building scene recognition based on multi-view visual information has become a crucial step in achieving high-precision 3D reconstruction and semantic understanding. This technology relies on extracting stable and matchable feature points from image sequences acquired from different spatial locations and restoring the structure through geometric constraints. However, in modern urban environments, many high-rise buildings use glass curtain walls as their facade material. The strong specular reflection properties of these glass curtain walls cause a large amount of false texture and pseudo-structural information to be mixed into the multi-view images captured by drones, seriously interfering with the reliability of traditional feature extraction and matching algorithms.

[0003] Feature matching processes based on classic methods such as Scale Invariant Feature Transform (SIFT) are prone to mismatches when dealing with reflective areas of glass curtain walls. The reflected content dynamically changes with the viewing angle, causing the same physical location to appear in drastically different visual patterns in different images, while adjacent non-reflective areas may be incorrectly associated due to mirror similarity. Such mismatches not only disrupt the geometric consistency of subsequent triangulation and pose estimation but also lead to point cloud holes, structural distortion, and even overall reconstruction failure. Although existing dehazing or dereflection algorithms attempt to suppress interference through image enhancement, they typically assume the degradation model is atmospheric scattering or uniform noise, failing to effectively distinguish between the virtual image generated by physical reflection and the true geometric structure of the building itself, especially vulnerable in scenarios with complex lighting and multiple reflections.

[0004] Existing technologies for handling multi-UAV collaborative vision tasks generally lack a deep integration of optical physical mechanisms and data-driven models. On the one hand, while methods based purely on deep learning can learn reflection patterns, their generalization ability is limited by the distribution of training data, making it difficult to handle unfamiliar building materials and lighting combinations. On the other hand, although traditional polarization imaging can partially separate reflection components from a single viewpoint, it is not effectively coupled with multi-view geometric constraints and collaborative perception frameworks, resulting in low information utilization. Therefore, in urban environments with dense high-reflectivity buildings, there is an urgent need for a novel visual processing mechanism that can suppress specular reflection interference at the physical level and deeply integrate with the multi-UAV collaborative recognition process to overcome the fundamental bottlenecks of existing methods in terms of feature robustness and structural realism. Summary of the Invention

[0005] The technical problem this invention aims to solve is that in existing methods for visual recognition of buildings using multiple drones, when facing highly specular reflective surfaces such as glass curtain walls, the clear reflections of environmental objects on the building surface cause traditional matching algorithms based on local feature descriptors, such as scale-invariant feature transform algorithms, to misidentify reflective areas with inconsistent content and location from different drone perspectives as structural feature points of the building itself, resulting in a large number of incorrect matching pairs. These incorrect matching pairs severely interfere with subsequent motion reconstruction and multi-view stereo matching processes, leading to severe geometric distortion, holes, or incorrect reconstructions in the final generated 3D model. Furthermore, existing image dehazing or image enhancement algorithms, whose physical models are designed to handle atmospheric scattering effects, cannot physically distinguish and handle specular reflection phenomena, and therefore cannot effectively eliminate the interference of reflection on feature matching.

[0006] To address the aforementioned technical issues, this invention provides a multi-UAV visual collaborative building scene recognition method. This method constructs a feature point hierarchical mechanism based on joint verification of multi-view geometric constraints and a photometric physical model. First, a UAV cluster synchronized with a spatiotemporal reference acquires multi-view image sequences of buildings and precise pose information corresponding to each frame. Second, the fundamental matrix between any image pair is directly calculated using the precise extrinsic parameters of the UAVs, thereby establishing strict epipolar geometric constraints. Then, all candidate feature points are extracted in the image domain without initial authenticity checks. The core step involves simultaneously performing dual checks on geometric consistency and photometric consistency for any set of candidate matching points spanning multiple viewpoints. The geometric consistency check uses epipolar constraints to determine whether the set of matching points converges to a stable three-dimensional point in space; the photometric consistency check introduces a physical reflection model based on Fresnel equations, analyzing the variation of brightness of the local image patch where the matching point is located with the observation viewpoint to determine whether it better conforms to diffuse reflection surface characteristics or specular reflection surface characteristics. Finally, based on the results of the double test, all candidate feature points were divided into a set of high-confidence structural feature points and a set of reflection feature points. Only the former was used for subsequent 3D reconstruction of the building structure, thereby fundamentally eliminating the interference caused by specular reflection and obtaining a high-precision 3D building model without geometric distortion.

[0007] According to an aspect of the present invention, a method for building scene recognition using multi-UAV visual collaboration is provided, comprising the following steps:

[0008] The system acquires a sequence of multi-view images collected when multiple UAVs conduct close-up observations of a target building along a preset flight path, and synchronously records the six-degree-of-freedom pose information of the UAVs at the moment of acquisition of each frame of image. The six-degree-of-freedom pose information includes three-dimensional spatial position coordinates and three-axis attitude angles.

[0009] Based on the six-degree-of-freedom pose information collected by the multi-UAV, the relative rotation matrix and translation vector between any two different viewpoint images are calculated, and the essential matrix and fundamental matrix describing the projective geometric relationship between the two viewpoints are constructed accordingly.

[0010] Each frame of the multi-view image sequence is processed to extract a set of candidate feature points. Each candidate feature point includes its two-dimensional coordinates in the image, the appearance descriptor of the local image region, and the initial confidence level.

[0011] Establish a cross-view candidate feature point matching graph, where the nodes of the graph are all candidate feature points in all images, and the edges of the graph represent the initial matching relationship calculated based on the appearance descriptor similarity.

[0012] An iterative feature point hierarchical and purification process based on multi-view geometry and photometric consistency joint verification is performed to divide the candidate feature points into a set of structural feature points and a set of reflection feature points.

[0013] Using only the feature points in the structural feature point set and their matching relationships, the motion recovery structure algorithm is executed to calculate the accurate camera parameters from all UAV perspectives and generate a sparse 3D point cloud model of the building.

[0014] Based on the sparse 3D point cloud model and accurate camera parameters, a multi-view stereo matching algorithm is used to perform dense 3D reconstruction of the building, generating the final refined 3D mesh model.

[0015] As one embodiment of the present invention, the acquisition of the six-degree-of-freedom pose information of multiple unmanned aerial vehicles specifically includes:

[0016] Each UAV is equipped with a Global Navigation Satellite System (GNSS) receiver integrating real-time dynamic differential technology and a high-precision inertial measurement unit (IMU). The GNSS receiver provides three-dimensional absolute position coordinates with centimeter-level accuracy, while the IMU provides the UAV's roll, pitch, and yaw angles relative to the geographic coordinate system. Synchronization signals transmitted from the ground control station ensure that all UAVs have a strictly consistent time reference, thereby achieving precise alignment of all pose data and image data in terms of timestamps.

[0017] As one embodiment of the present invention, the extraction of a set of candidate feature points specifically includes:

[0018] For each frame of the image, the Laplacian operator is first applied for edge detection, and combined with the local entropy value of the image, the information-rich regions in the image are initially identified. Within these regions, the accelerated robust feature algorithm is used for corner detection to generate the positions of candidate feature points. Around the position of each candidate feature point, a fixed-size image patch is extracted, and the binary string is calculated as its appearance descriptor using the binary robust independent element feature algorithm.

[0019] As one embodiment of the present invention, the iterative feature point layering and purification process based on multi-view geometry and photometric consistency joint verification specifically includes:

[0020] In the initialization step, all candidate feature points are placed into the set of undetermined feature points;

[0021] The process involves iterative looping, in which an initial matching cluster consisting of candidate feature points from at least three different perspectives is randomly selected from the set of undetermined feature points in each iteration.

[0022] The initial matching cluster is subjected to geometric consistency verification, specifically as follows: select any two feature points in the matching cluster, and use a pre-calculated fundamental matrix to calculate the algebraic distance from one point to the epipolar line corresponding to the other point; if the distance is greater than a preset geometric error threshold, the matching cluster is determined to be geometrically inconsistent; if the geometric error of all point pairs in the matching cluster is less than the threshold, the coordinates of the three-dimensional spatial points are calculated by triangulation, and the spatial distance from the feature points in all viewpoints to their corresponding three-dimensional spatial point back projection rays is calculated; if the root mean square error of all distances is less than a preset three-dimensional reprojection error threshold, the matching cluster is determined to be geometrically consistent.

[0023] For the matching clusters that pass the geometric consistency check, a further photometric consistency check is performed. Specifically, based on the 3D spatial coordinates of the point obtained by triangulation, the preliminary normal vector estimate of the building surface where the point is located, and the camera position from each UAV viewpoint, a half-angle vector between the incident ray and the camera line of sight is calculated for each viewpoint. According to the Fresnel physical model, the intensity of specular reflection is closely related to the angle between the half-angle vector and the surface normal vector, while the intensity of diffuse reflection is independent of the viewpoint. The average brightness value of the local image patch of all feature points in the matching cluster is calculated, and the trend of the brightness value of this group changes with the viewpoint is analyzed. This trend is fitted with the specular reflection brightness change trend predicted by the Fresnel model and the constant brightness trend of the ideal diffuse reflection model to obtain the photometric model conformity index.

[0024] Based on the results of the geometric consistency check and the photometric consistency check, the feature points in the matching cluster are assigned to the following categories: if the geometry is inconsistent, all feature points in the matching cluster are marked as invalid and removed from the undetermined set; if the geometry is consistent and the photometric model conformity index shows that it conforms more to the diffuse reflection model, all feature points in the matching cluster are moved from the undetermined set to the structural feature point set; if the geometry is consistent and the photometric model conformity index shows that it conforms more to the specular reflection model, all feature points in the matching cluster are moved from the undetermined set to the reflection feature point set.

[0025] Repeat the iterative loop process until the set of undetermined feature points is empty, thus completing the stratification of all feature points.

[0026] As one embodiment of the present invention, the preliminary normal vector estimation in the photometric consistency verification specifically includes:

[0027] Before performing photometric consistency verification, an initial sparse point cloud is constructed using the three-dimensional spatial points corresponding to all matching clusters that have passed geometric consistency verification. Plane fitting is performed on the initial sparse point cloud to identify planar fragments representing the main facade of the building. For the three-dimensional spatial points corresponding to the matching clusters to be verified, they are assigned to the nearest planar fragment, and the normal vector of the planar fragment is used as its initial surface normal vector.

[0028] As one embodiment of the present invention, the execution of the motion recovery structure algorithm specifically includes:

[0029] An incremental motion-based structure recovery process is adopted. First, a pair of images with a sufficient number of matching structural feature points and a suitable baseline length are selected as the initial image pair. Robust fundamental matrix estimation and triangulation are performed to generate an initial 3D point cloud. Subsequently, new viewpoint images are iteratively selected, and their camera poses are solved using a perspective orthogonal projection algorithm. Structural feature points newly observed from this viewpoint that match points in the existing 3D point cloud are added to the 3D point cloud through triangulation. After each addition of a new viewpoint or point, global bundle adjustment optimization is performed. At the same time, nonlinear least squares optimization is performed on the spatial coordinates of all camera poses and all 3D points to minimize the error between the observed positions of all structural feature points and the back-projected positions of their corresponding 3D points on the image.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. This invention proposes a novel technical solution to address the root cause of specular reflection interference on building surfaces. Its core lies not in attempting image repair or enhancement, but in establishing a rigorous dual physical and geometric verification mechanism to proactively identify and stratify the attribute of feature points in the image. By introducing epipolar geometric constraints based on the precise pose of a UAV, this method can effectively eliminate false feature points with inconsistent positions in three-dimensional space, representing the first level of solving the reflection problem.

[0032] 2. This method innovatively introduces a photometric physical model based on the Fresnel equation. By analyzing the variation of feature point brightness with viewing angle, its physical causes are determined, enabling accurate differentiation between geometrically consistent feature points with different physical causes (i.e., real structural features on the glass surface and spurious environmental features reflected from the glass surface). This dual verification mechanism clearly divides candidate feature points into a set of structural feature points and a set of reflection feature points. This ensures that the subsequent 3D reconstruction process uses only rigorously selected high-quality data, completely avoiding the contamination of model accuracy by reflection artifacts. As a result, it can generate architectural 3D models with extremely accurate geometry and complete surface details, demonstrating significant technical effectiveness. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall technical solution architecture of a multi-UAV visual collaborative building scene recognition method proposed in this invention;

[0034] Figure 2 This is a schematic diagram of the core principle framework of the feature point hierarchical mechanism based on the joint verification of multi-view geometric constraints and photometric physical model in this invention;

[0035] Figure 3 This is a flowchart illustrating the logical flow of the multi-view image acquisition and pose synchronization stage in this invention.

[0036] Figure 4 This is a logical flowchart of the candidate feature point extraction and cross-view matching map construction stages in this invention. Detailed Implementation

[0037] Please refer to Figures 1 to 4This invention provides a multi-UAV visual collaborative building scene recognition method, aiming to solve the SIFT feature point mismatch problem caused by highly specular reflective surfaces such as glass curtain walls in multi-view images. This problem stems from the clear reflections of environmental objects on building surfaces, causing traditional matching algorithms based on local feature descriptors to misclassify reflective areas with inconsistent content and location as structural feature points of the building itself. This results in a large number of incorrect matching pairs, severely interfering with motion reconstruction and multi-view stereo matching processes, ultimately leading to geometric distortion, holes, or incorrect reconstructions in the 3D model. Existing image dehazing or enhancement algorithms are based on atmospheric scattering physics models, which cannot physically distinguish and process specular reflection phenomena, thus failing to effectively eliminate the interference of reflection on feature matching. This embodiment constructs a feature point hierarchical mechanism based on joint verification of multi-view geometric constraints and photometric physics models, fundamentally eliminating the interference caused by specular reflection and obtaining a high-precision, geometrically distortion-free 3D building model.

[0038] The method first acquires a sequence of multi-view images collected by multiple UAVs conducting close-range observations of a target building along a preset flight path, and simultaneously records the six-degree-of-freedom pose information of the UAV at the moment of acquisition for each frame. Each UAV is equipped with a Global Navigation Satellite System (GNSS) receiver integrating real-time dynamic differential technology and a high-precision inertial measurement unit (IMU). The GNSS receiver provides centimeter-level accuracy in three-dimensional absolute position coordinates, while the IMU provides the UAV's roll, pitch, and yaw angles relative to the geographic coordinate system. The ground control station transmits a synchronization signal to ensure strict consistency of the time reference for all UAVs, achieving precise alignment of all pose data and image data in terms of timestamps. This synchronization mechanism ensures that any two frames of images from different UAVs can be correlated with their precise spatial poses, laying the foundation for subsequent geometric constraint calculations.

[0039] Based on the acquired six-degree-of-freedom pose information, calculate the relative rotation matrix and translation vector between any two images from different viewpoints. Let the rotation matrix of the camera coordinate system in the first viewpoint relative to the world coordinate system be... The translation vector is The corresponding parameters for the second-person perspective are: , Then the relative rotation matrix between the two viewpoints Relative translation vector Based on this, and combined with the camera intrinsic parameter matrix K, the essential matrix is ​​constructed. ,in express The antisymmetric matrix; and then the fundamental matrix. This fundamental matrix fully describes the projective geometric relationship between the two viewpoint images, providing a mathematical basis for subsequent epipolar constraints.

[0040] Each frame of the multi-view image sequence is processed to extract a set of candidate feature points. The processing includes: first, applying the Laplacian operator for edge detection, calculating the second derivative response of each pixel to highlight high-frequency detail areas; simultaneously, calculating the local entropy value of the image, i.e., the information entropy of the gray-level distribution within a fixed window centered on each pixel, used to measure local texture complexity. Regions with Laplacian response intensity higher than a preset threshold and local entropy values ​​greater than another preset threshold are marked as information-rich candidate regions. Within these regions, an accelerated robust feature algorithm is used for corner detection. This algorithm compares the brightness difference between the central pixel and its surrounding annular sampling points to select stable corners with significant local contrast as candidate feature point locations. Around the location of each candidate feature point, a fixed-size 16×16 pixel image patch is extracted, and a 128-bit binary string is calculated using a binary robust independent element feature algorithm as its appearance descriptor. This descriptor is generated by encoding the brightness relationship of specific pixel pairs within the image patch, possessing good scale and rotation invariance, and is suitable for cross-view matching.

[0041] A cross-view candidate feature point matching graph is constructed, where the nodes of the graph represent all candidate feature points in all images, and the edges represent the initial matching relationships calculated based on appearance descriptor similarity. Specifically, for any two candidate feature points in images from different viewpoints, the Hamming distance between their binary descriptors is calculated; if this distance is less than a preset matching threshold, an edge is established between the corresponding two nodes, indicating that they constitute an initial matching pair. This matching graph covers all viewpoints, forming a global feature association network, providing a data foundation for subsequent joint verification.

[0042] An iterative feature point hierarchical and purification process based on joint verification of multi-view geometry and photometric consistency is performed, dividing candidate feature points into a structural feature point set and a reflection feature point set. This process includes the following sub-steps:

[0043] In the initialization step, all candidate feature points are placed into the set of undetermined feature points. This set initially contains all extracted and preliminarily matched feature points, but has not yet undergone true / false determination.

[0044] The process involves iterative looping. In each iteration, an initial matching cluster is randomly selected from the set of undetermined feature points, consisting of candidate feature points from at least three different viewpoints. The construction of this matching cluster is based on the aforementioned matching map, ensuring that there are pairwise matching relationships between the selected feature points, and that each feature point comes from three or more independent viewpoints, thus satisfying the minimum observation conditions required for triangulation.

[0045] Perform geometric consistency verification on the initial matching cluster. Specifically, select any two feature points from the matching cluster. and , respectively located in the Perspective and the first In the viewpoint image. Using the pre-calculated first... Perspective and the first The fundamental matrix between perspectives Calculation points Time Corresponding polar lines algebraic distance If the distance is greater than the preset geometric error threshold... (Typical value is 1.5 pixels), then the matching cluster is determined to be geometrically inconsistent. If the geometric error of all point pairs in the matching cluster is less than 1.5 pixels, then the matching cluster is considered geometrically inconsistent. If the cluster satisfies the epipolar constraint, the next step is triangulation. The linear equations are solved using singular value decomposition to calculate the coordinates P of the point in three-dimensional space. Subsequently, feature points from all viewpoints are calculated. The spatial distance to the back-projected ray from its corresponding three-dimensional point P. This distance is defined as the distance from the point P to the back-projected ray. Rather than in the Points obtained by backprojection of P on the view image The Euclidean distance between them. Calculate the root mean square error of all such distances. ,in This is the total number of feature points in the matching cluster. If Less than the preset 3D reprojection error threshold (Typical value is 0.8 pixels), then the matching cluster is determined to be geometrically consistent; otherwise, it is determined to be geometrically inconsistent.

[0046] For matching clusters that pass the geometric consistency check, a further photometric consistency check is performed. The core of this step lies in introducing a physical reflection model based on Fresnel equations to distinguish between diffuse and specular reflection characteristics. First, based on the triangulated three-dimensional spatial coordinates P of the point and the preliminary normal vector estimation of the building surface where that point is located... and the camera positions from the perspective of each drone. Calculate the half-angle vector between the incident ray and the camera's line of sight for each viewpoint. The incident ray direction is defined as from P towards the ambient light source. However, in the absence of explicit light source information, it can be assumed that the ambient light is uniformly diffused, in which case specular reflection is mainly contributed by strong reflections from the sky or surrounding buildings. A more practical approach is to use a half-angle vector... Defined as pointing from P to the first View camera position Direction vector With surface normal vector The direction of the angle bisector, i.e. According to the Fresnel model of physics, the intensity of specular reflection... with half-angle vector and surface normal vector The included angle They are closely related, and their relationship can be approximated as follows:

[0047]

[0048] in , These are the refractive indices of the two media, respectively. Angle of incidence For the air-glass interface, [the angle of refraction is missing]. ≈1.0, ≈1.5. This formula shows that the reflection intensity increases sharply when the viewing angle is close to the direction of specular reflection. In contrast, the intensity of the ideal diffuse reflection model (Lambertian model) is... It is independent of perspective and remains constant.

[0049] In practical calculations, it is unnecessary to precisely solve the complex formulas mentioned above. Instead, the physical cause is determined by analyzing the variation trend of the average brightness value of the local image patch of all feature points in the matching cluster with the viewing angle. Specifically, the average gray value of the 16×16 pixel image patch corresponding to each feature point is extracted. Index the viewpoint As the independent variable, As the dependent variable, a function is fitted with respect to changes in viewing angle. Simultaneously, the camera position at each viewing angle is considered. With surface normal vector Calculate the viewpoint and The included angle If the feature points originate from specular reflection, then Should be in It reaches its peak value when approaching a certain specific value (corresponding to the specular reflection angle), exhibiting a clear non-monotonic change; if it originates from diffuse reflection, then... It should remain basically stable. Therefore, the photometric model conformity index is calculated. Defined as the measured brightness sequence { } and the ideal diffuse constant sequence { Pearson correlation coefficient And the specular reflection sequence predicted by the Fresnel model { Correlation coefficient .like ( If the confidence margin is set to a preset value (typically 0.2), it is determined to be more consistent with the diffuse reflection model; otherwise, it is determined to be a specular reflection model.

[0050] The specific implementation of the preliminary normal vector estimation is as follows: Before performing photometric consistency verification, an initial sparse point cloud is constructed using the 3D spatial points corresponding to all matching clusters that have passed geometric consistency verification. A region growing plane fitting algorithm is then executed on this initial sparse point cloud. The algorithm uses each point as a seed to search for points in its neighborhood that satisfy the coplanarity condition (the distance from the point to the fitting plane is less than a threshold), forming planar fragments. Principal component analysis is used to calculate the optimal fitting plane for each fragment, and its normal vector is used as the representative direction of that fragment. For the 3D spatial point P corresponding to the matching cluster to be verified, its distance to all identified planar fragments is calculated, and it is assigned to the nearest planar fragment. The normal vector of this planar fragment is used as the preliminary surface normal vector n of P. Although this estimation is preliminary, it is sufficient to support the half-angle vector calculation required for photometric consistency verification.

[0051] Based on the results of geometric consistency and photometric consistency checks, feature points in a matching cluster are assigned to specific locations: if geometrically inconsistent, all feature points in that cluster are marked as invalid and removed from the pending set; if geometrically consistent and the photometric model conformity index shows a stronger fit to the diffuse reflection model, all feature points in that cluster are moved from the pending set to the structural feature point set; if geometrically consistent and the photometric model conformity index shows a stronger fit to the specular reflection model, all feature points in that cluster are moved from the pending set to the reflective feature point set. This decision logic ensures that only feature points that simultaneously satisfy both geometric stability and physical realism can enter the structural feature point set.

[0052] The iterative process is repeated until the set of undetermined feature points is empty, completing the stratification of all feature points. This iterative process employs a random sampling strategy to ensure that all possible matching clusters have a chance to be checked, avoiding omissions.

[0053] This algorithm utilizes only feature points in a set of structural feature points and their matching relationships to perform a motion-to-structure reconstruction algorithm to calculate accurate camera parameters for all UAV perspectives and generate a sparse 3D point cloud model of the building. The algorithm employs an incremental process: first, it selects a pair of images with a sufficient number of structural feature point matching pairs (no fewer than 50 pairs) and a suitable baseline length (10% to 30% of the building width) as initial image pairs. Robust fundamental matrix estimation is performed on these image pairs, and an eight-point algorithm combined with RANSAC is used to remove residual outlier matches. Linear triangulation is then performed to generate the initial 3D point cloud. Next, new viewpoint images are iteratively selected, and their camera poses are solved using a perspective orthogonal projection algorithm. The algorithm uses structural feature points in the new images that match points in the existing 3D point cloud to construct a reprojection error equation, and then uses Levenberg-Marquardt optimization to solve for the optimal camera extrinsic parameters. Newly observed structural feature points from this viewpoint that match points in the existing 3D point cloud are added to the 3D point cloud through triangulation. Global bundle adjustment is performed after each addition of a new viewpoint or point. This optimization applies to all camera poses simultaneously. and the spatial coordinates of all three-dimensional points { The nonlinear least squares optimization is performed, with the objective function being to minimize the observed positions of all structural feature points. Its corresponding three-dimensional point P j Backprojection position on the image The sum of the squared errors between them, that is:

[0054]

[0055] in This represents the camera projection function. This global optimization significantly improves the accuracy and consistency of camera parameters and 3D point coordinates.

[0056] Based on a sparse 3D point cloud model and precise camera parameters, a multi-view stereo matching algorithm is used to perform dense 3D reconstruction of buildings, generating a final refined 3D mesh model. This algorithm uses the sparse point cloud as a guide, performing dense matching between each pair of adjacent view images. The matching cost calculation comprehensively considers color consistency, gradient consistency, and depth priors based on the sparse point cloud. A semi-global matching strategy is used to aggregate costs along multiple directions to generate an initial depth map. Subsequently, depth maps from all views are fused to remove outlier depth values, generating a unified point cloud. Finally, a Poisson surface reconstruction algorithm is used to convert the point cloud into a continuous triangular mesh model, and normal vector smoothing and hole repair are performed to output the final 3D building model.

[0057] Throughout the methodology, data flow and processing are highly coordinated. The image acquisition and pose synchronization module outputs timestamped image frames and pose quadruples; the geometric relationship calculation module receives pose data and outputs the fundamental matrix of all image pairs; the feature extraction module processes single-frame images and outputs a candidate point list with descriptors; the matching map construction module integrates all candidate points and outputs global matching relationships; the feature layering module, as the core, receives the matching map, fundamental matrix, and image data, and outputs a set of structural feature points and a set of reflection feature points; the reconstruction module receives the set of structural feature points, performs sparse reconstruction and dense reconstruction sequentially, and outputs a 3D mesh model. All modules communicate through standardized data interfaces to ensure the robustness and scalability of the process.

[0058] An anomaly handling mechanism is implemented throughout the entire process. During pose synchronization, if a UAV loses its Global Navigation Satellite System (GNSS) signal, dead reckoning by the inertial measurement unit (INS) is activated, and trajectory smoothing correction is performed after the signal is restored. During feature extraction, if a local area of ​​the image is overexposed or underexposed, the feature point extraction density for that area is reduced or skipped entirely. During geometric verification, if the triangulation result is unstable (e.g., negative or excessive depth), it is directly determined to be geometrically inconsistent. During photometric verification, if surface normal vector estimation fails (e.g., point P is far from all planar fragments), the matching cluster is temporarily shelved until subsequent sparse point cloud updates. During reconstruction, if there are too few structural feature points in an image from a certain viewpoint, that viewpoint is skipped to avoid introducing noise.

[0059] This embodiment achieves high-precision identification and reconstruction of glass curtain wall building scenes through the above detailed steps, effectively overcoming the technical obstacles caused by specular reflection.

[0060] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for building scene recognition using multi-UAV visual collaboration, characterized in that, include: The system acquires a sequence of multi-view images collected by multiple UAVs on a preset flight path for close-up observation of a target building, and synchronously records the six-degree-of-freedom pose information of the UAVs at the moment of acquisition of each frame of image. The six-degree-of-freedom pose information includes three-dimensional spatial position coordinates and three-axis attitude angles. Based on the six-degree-of-freedom pose information, the relative rotation matrix and translation vector between any two different viewpoint images are calculated, and the essential matrix and fundamental matrix describing the projective geometric relationship between the two viewpoints are constructed accordingly. Each frame of the multi-view image sequence is processed to extract a set of candidate feature points. Each candidate feature point includes its two-dimensional coordinates in the image, the appearance descriptor of the local image region, and the initial confidence level. Establish a cross-view candidate feature point matching graph, where the nodes of the graph are all candidate feature points in all images, and the edges of the graph represent the initial matching relationship calculated based on the appearance descriptor similarity. An iterative feature point hierarchical and purification process based on multi-view geometry and photometric consistency joint verification is performed to divide the candidate feature points into a set of structural feature points and a set of reflection feature points. Using only the feature points in the structural feature point set and their matching relationships, the motion recovery structure algorithm is executed to calculate the accurate camera parameters from all UAV perspectives and generate a sparse 3D point cloud model of the building. Based on the sparse 3D point cloud model and accurate camera parameters, a multi-view stereo matching algorithm is used to perform dense 3D reconstruction of the building, generating the final refined 3D mesh model. An iterative feature point hierarchical and purification process based on multi-view geometric and photometric consistency joint verification is performed. This also includes further photometric consistency verification of matching clusters that pass the geometric consistency verification, specifically: Based on the three-dimensional spatial coordinates of the point obtained by triangulation, the preliminary normal vector estimate of the building surface where the point is located, and the camera position from each UAV perspective, the half-angle vector between the incident ray and the camera line of sight is calculated for each perspective. According to the Fresnel physics model, the intensity of specular reflection is closely related to the angle between the half-angle vector and the surface normal vector, while the intensity of diffuse reflection is independent of the viewing angle. Calculate the average brightness value of the local image patch of all feature points in the matching cluster, and analyze the trend of brightness value change with viewing angle; The trend is fitted with the specular reflection brightness change trend predicted by the Fresnel model and the constant brightness trend of the ideal diffuse reflection model to obtain the photometric model fit index. Based on the results of the geometric consistency check and the photometric consistency check, the feature points in the matching cluster are assigned to a specific category: If the geometry is inconsistent, all feature points in the matching cluster are marked as invalid and removed from the undetermined set; if the geometry is consistent and the photometric model conformance index shows that it conforms more to the diffuse reflection model, all feature points in the matching cluster are moved from the undetermined set to the structural feature point set; if the geometry is consistent and the photometric model conformance index shows that it conforms more to the specular reflection model, all feature points in the matching cluster are moved from the undetermined set to the reflection feature point set. Repeat the iterative process until the set of undetermined feature points is empty, thus completing the stratification of all feature points.

2. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, Acquire six-DOF pose information of multiple UAVs, including: Each UAV is equipped with a Global Navigation Satellite System (GNSS) receiver integrating real-time dynamic differential technology and a high-precision inertial measurement unit (IMU). The GNSS receiver provides three-dimensional absolute position coordinates with centimeter-level accuracy, while the IMU provides the UAV's roll, pitch, and yaw angles relative to the geographic coordinate system. Synchronization signals transmitted from the ground control station ensure that all UAVs have a strictly consistent time reference, thereby achieving precise alignment of all pose data and image data in terms of timestamps.

3. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, Extract a set of candidate feature points, including: For each frame of the image, the Laplacian operator is first applied for edge detection, and combined with the local entropy value of the image, the information-rich regions in the image are initially identified. Within these regions, the accelerated robust feature algorithm is used for corner detection to generate the positions of candidate feature points. Around the position of each candidate feature point, a fixed-size image patch is extracted, and the binary string is calculated as its appearance descriptor using the binary robust independent element feature algorithm.

4. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, An iterative feature point hierarchical and purification process based on joint verification of multi-view geometry and photometric consistency is performed, including: In the initialization step, all candidate feature points are placed into the set of undetermined feature points; The process involves iterative looping, in which an initial matching cluster composed of candidate feature points from at least three different perspectives is randomly selected from the set of undetermined feature points in each iteration.

5. The multi-UAV visual collaborative building scene recognition method according to claim 4, characterized in that, An iterative feature point layering and purification process based on multi-view geometry and photometric consistency joint verification is performed. This process also includes geometric consistency verification of the initial matching cluster. Specifically, this involves selecting any two feature points in the matching cluster and calculating the algebraic distance from one point to the epipolar line corresponding to the other point using a pre-calculated fundamental matrix. If this distance is greater than a preset geometric error threshold, the matching cluster is determined to be geometrically inconsistent. If the geometric errors of all point pairs in the matching cluster are less than the threshold, the coordinates of the three-dimensional spatial points are calculated using a triangulation method, and the spatial distances from the feature points in all viewpoints to their corresponding three-dimensional spatial backprojection rays are calculated. If the root mean square error of all distances is less than a preset three-dimensional reprojection error threshold, the matching cluster is determined to be geometrically consistent.

6. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, The preliminary normal vector estimation in the photometric consistency verification includes: Before performing photometric consistency verification, an initial sparse point cloud is constructed using the three-dimensional spatial points corresponding to all matching clusters that have passed geometric consistency verification. Plane fitting is performed on the initial sparse point cloud to identify planar fragments representing the main facade of the building. For the three-dimensional spatial points corresponding to the matching clusters to be verified, they are assigned to the nearest planar fragment, and the normal vector of the planar fragment is used as its initial surface normal vector.

7. The multi-UAV visual collaborative building scene recognition method according to claim 5, characterized in that, In the geometric consistency check of the initial matching cluster, the algebraic distance is calculated as follows: for the first... Feature points in a viewpoint image With the Feature points in a viewpoint image , using the Perspective and the first The fundamental matrix between perspectives Calculation points Time Corresponding polar lines algebraic distance .

8. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, The photometric model conformity index is calculated as follows: the Pearson correlation coefficient between the measured luminance sequence and the ideal constant diffuse reflectance sequence is calculated. And the correlation coefficient with the specular reflection sequence predicted by the Fresnel model. ;like If it is more consistent with the diffuse reflection model, then it is determined that it better fits the diffuse reflection model. To pre-set the confidence margin.

9. The multi-UAV visual collaborative building scene recognition method according to claim 1, characterized in that, Execute the motion recovery structure algorithm, including: An incremental motion-based structure recovery process is adopted. First, a pair of images with a sufficient number of matching structural feature points and a suitable baseline length are selected as the initial image pair. Robust fundamental matrix estimation and triangulation are performed to generate an initial 3D point cloud. Subsequently, new viewpoint images are iteratively selected, and their camera poses are solved using a perspective orthogonal projection algorithm. Structural feature points newly observed from this viewpoint that match points in the existing 3D point cloud are added to the 3D point cloud through triangulation. After each addition of a new viewpoint or point, global bundle adjustment optimization is performed. At the same time, nonlinear least squares optimization is performed on the spatial coordinates of all camera poses and all 3D points to minimize the error between the observed positions of all structural feature points and the back-projected positions of their corresponding 3D points on the image.

Images