Methods and systems for fusing satellite imagery and UAV SLAM data

Abstract

This invention discloses a method and system for fusing satellite imagery and UAV SLAM data, relating to the field of image processing technology. The method includes: constructing a first construction monitoring model and a second construction monitoring model; spatially registering and fusing the first and second construction monitoring models to construct a third construction monitoring model; constructing a site feature registration constraint space, and using this space to perform multimodal feature registration detection and correction on the third construction monitoring model to obtain a fourth construction monitoring model; performing multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model to obtain a model unfolding detection anomaly tracing map; and performing adaptive correction on the fourth construction monitoring model to obtain a fifth construction monitoring model. This invention solves the technical problem of low accuracy in the fusion of satellite imagery and UAV SLAM data in existing technologies, achieving the technical effect of improving the accuracy of satellite imagery and UAV SLAM data fusion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for fusing satellite imagery and UAV SLAM data. Background Technology

[0002] In practical applications of 3D modeling at construction sites, the fusion of satellite imagery and UAV SLAM data often yields unsatisfactory results. While satellite data provides a broad macroscopic view, its spatial resolution is limited, making it difficult to capture the fine structures of the construction site. Although UAV SLAM can generate high-precision local models, it is susceptible to dynamic environmental interference, resulting in cumulative errors. These factors make it difficult to achieve high accuracy and consistency when fusing satellite imagery and UAV SLAM data, thus affecting the efficiency and accuracy of construction monitoring and management. Summary of the Invention

[0003] This application provides a method and system for fusing satellite imagery and UAV SLAM data, which is used to address the technical problem of low accuracy in the fusion of satellite imagery and UAV SLAM data in the prior art.

[0004] In view of the above problems, this application provides a method and system for fusing satellite imagery and UAV SLAM data.

[0005] A first aspect of this application provides a method for fusing satellite imagery and UAV SLAM data, the method comprising:

[0006] Real-time monitoring of the construction site is conducted using remote sensing satellite systems and UAV SLAM systems to construct a first construction monitoring model and a second construction monitoring model. A unified coordinate system is established, and the first and second construction monitoring models are spatially registered and fused according to the unified coordinate system to construct a third construction monitoring model. Multimodal feature registration constraint analysis is performed on the first and second construction monitoring models to construct a site feature registration constraint space. Multimodal feature registration detection and correction are performed on the third construction monitoring model according to the site feature registration constraint space to obtain a fourth construction monitoring model. Multi-scale unfolding detection and anomaly tracing are performed on the fourth construction monitoring model to obtain a model unfolding detection anomaly tracing map. Adaptive correction is performed on the fourth construction monitoring model according to the model unfolding detection anomaly tracing map to obtain a fifth construction monitoring model.

[0007] A second aspect of this application provides a system for fusing satellite imagery and UAV SLAM data, the system comprising:

[0008] The system comprises the following modules: a real-time monitoring module for real-time monitoring of the construction site using remote sensing satellite systems and UAV SLAM systems, constructing a first construction monitoring model and a second construction monitoring model; a spatial registration and fusion module for constructing a unified coordinate system and performing spatial registration and fusion of the first and second construction monitoring models based on this unified coordinate system, constructing a third construction monitoring model; a registration constraint analysis module for performing multimodal feature registration constraint analysis on the first and second construction monitoring models, constructing a site feature registration constraint space; a registration detection and correction module for performing multimodal feature registration detection and correction on the third construction monitoring model based on the site feature registration constraint space, obtaining a fourth construction monitoring model; an anomaly tracing module for performing multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model, obtaining a model unfolding detection anomaly tracing map; and an adaptive correction module for performing adaptive correction on the fourth construction monitoring model based on the model unfolding detection anomaly tracing map, obtaining a fifth construction monitoring model.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] This application utilizes remote sensing satellite systems and UAV SLAM systems to monitor construction sites in real time, constructing a first construction monitoring model and a second construction monitoring model. A unified coordinate system is established, and the first and second construction monitoring models are spatially registered and fused according to this unified coordinate system to construct a third construction monitoring model. Multimodal feature registration constraint analysis is performed on the first and second construction monitoring models to construct a site feature registration constraint space. Multimodal feature registration detection and correction are performed on the third construction monitoring model according to the site feature registration constraint space to obtain a fourth construction monitoring model. Multi-scale unfolding detection and anomaly tracing are performed on the fourth construction monitoring model to obtain a model unfolding detection anomaly tracing map. Adaptive correction is performed on the fourth construction monitoring model according to the model unfolding detection anomaly tracing map to obtain a fifth construction monitoring model. This invention solves the technical problem of low accuracy in the fusion of satellite imagery and UAV SLAM data in the prior art. Through multi-source data fusion and precise registration and correction, it achieves the technical effect of improving the accuracy of satellite imagery and UAV SLAM data fusion. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the method for fusing satellite imagery and UAV SLAM data provided in an embodiment of this application;

[0013] Figure 2 A schematic diagram of the system structure for fusing satellite imagery and UAV SLAM data provided in this application embodiment.

[0014] Figure labeling: Real-time monitoring module 11, Spatial registration fusion module 12, Registration constraint parsing module 13, Registration detection and correction module 14, Anomaly tracing module 15, Adaptive correction module 16. Detailed Implementation

[0015] This application provides a method and system for fusing satellite imagery and UAV SLAM data, addressing the technical problem of low fusion accuracy of satellite imagery and UAV SLAM data in existing technologies. By fusing multi-source data and precise registration and correction, it achieves the technical effect of improving the fusion accuracy of satellite imagery and UAV SLAM data.

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0017] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0018] Example 1, as Figure 1 As shown, this application provides a method for fusing satellite imagery and UAV SLAM data, the method comprising:

[0019] Step S100: Real-time monitoring of the construction site is carried out using remote sensing satellite systems and UAV SLAM systems to construct a first construction monitoring model and a second construction monitoring model.

[0020] In this embodiment, when real-time monitoring of the construction site is conducted using a remote sensing satellite system and an unmanned aerial vehicle (UAV) SLAM system, the construction site is first continuously observed using the remote sensing satellite system to acquire satellite image monitoring data. Then, the satellite image monitoring data is multidimensionally cleaned based on satellite image cleaning factors including geometric correction, atmospheric correction, and radiometric correction to obtain on-site satellite image data. Next, an on-site satellite image model is constructed based on the on-site satellite image data. Finally, by comparing and correcting deviations in the on-site satellite image model, a first construction monitoring model is generated.

[0021] Simultaneously, the UAV SLAM system acquires SLAM datasets through real-time scanning of the construction site and constructs a site reconstruction model. Subsequently, point features are extracted from the SLAM dataset using the ORB-SLAM algorithm to obtain a set of point feature vectors, and line features are extracted using the LSD algorithm to obtain a set of line feature vectors. Next, joint matching is performed based on the point and line feature vector sets to construct multiple point-line joint constraint vectors. Finally, the site reconstruction model is jointly corrected based on these multiple point-line joint constraint vectors to obtain a second construction monitoring model.

[0022] Furthermore, the method provided in the application embodiments, which involves real-time monitoring of the construction site based on a remote sensing satellite system and an unmanned aerial vehicle (UAV) SLAM system, also includes:

[0023] The construction site is monitored in real time by the remote sensing satellite system to obtain satellite image monitoring data; the satellite image monitoring data is cleaned in multiple dimensions according to the satellite image cleaning factor to obtain on-site satellite image data; an on-site satellite image model is constructed based on the on-site satellite image data; the on-site satellite image model is compared and corrected for deviations based on the on-site satellite image data to generate the first construction monitoring model.

[0024] Furthermore, the method provided in the application embodiments also includes:

[0025] The satellite image cleaning factors include geometric correction, atmospheric correction, radiometric correction, image enhancement, and noise suppression.

[0026] In this embodiment of the application, the target construction site is first periodically photographed in real time using a remote sensing satellite system to obtain satellite image monitoring data.

[0027] Next, the satellite image monitoring data undergoes multi-dimensional cleaning based on satellite image cleaning factors. These factors include geometric correction, atmospheric correction, radiometric correction, image enhancement, and noise suppression. During geometric correction, the spatial position of the image is calibrated using ground control points and high-precision topographic data to eliminate geometric distortions. Then, atmospheric correction is performed, using an atmospheric radiative transfer model to compensate for atmospheric effects. Following this, radiometric correction is applied to uniformly adjust the image's radiometric values, ensuring consistency. Next, image enhancement is performed, adjusting contrast and brightness to enhance image detail. Finally, noise suppression, such as mean filtering or median filtering, removes noise and improves image quality. This process yields the on-site satellite image data.

[0028] Subsequently, a field satellite image model was constructed based on the on-site satellite image data. In this process, a stereo image pair dense matching algorithm was first used to extract and match feature points from the on-site satellite image data, generating a digital surface model. Then, the Delaunay triangulation algorithm was used to convert the digital surface model into a triangular mesh structure, establishing a three-dimensional geometric framework. Finally, the on-site satellite image data was used as a texture source, and texture mapping technology was used to precisely apply the image color information to the triangular mesh surface, forming the on-site satellite image model.

[0029] Finally, the on-site satellite image model was compared and corrected based on the on-site satellite image data. In this process, spatial location differences were identified through feature matching algorithms, and deviation correction parameters were calculated using least squares adjustment algorithms. The vertex coordinates of the on-site satellite image model were then adjusted to generate the first construction monitoring model.

[0030] Furthermore, the method provided in the application embodiments, which involves real-time monitoring of the construction site based on a remote sensing satellite system and an unmanned aerial vehicle (UAV) SLAM system, also includes:

[0031] The construction site is monitored in real time by the UAV SLAM system to obtain a SLAM dataset. A site reconstruction model is constructed based on the SLAM dataset. Point features are extracted from the SLAM dataset using the ORB-SLAM algorithm to obtain a set of point feature vectors. Line features are extracted from the SLAM dataset using the LSD algorithm to obtain a set of line feature vectors. Multiple point-line joint constraint vectors are constructed through joint matching of the point and line feature vector sets. The site reconstruction model is then jointly corrected based on these multiple point-line joint constraint vectors to obtain the second construction monitoring model.

[0032] In this embodiment, the UAV SLAM system first monitors the construction site in real time, acquiring continuous frame image sequences, 3D point cloud data, and motion data through its onboard visual sensors, LiDAR, and inertial measurement unit, thus obtaining a SLAM dataset containing image information, point cloud information, and pose information. Then, based on the SLAM dataset, visual odometry technology and point cloud registration algorithms are used to process the image sequences and point cloud data. An initial map is constructed through feature point matching and iterative nearest-point algorithms, and an incremental map construction method is used to gradually expand the environmental model, generating a site reconstruction model.

[0033] Subsequently, point features were extracted from the SLAM dataset using the ORB-SLAM algorithm. The ORB feature detector detected FAST corner points in the image and calculated their BRIEF descriptors, obtaining a set of point feature vectors representing local image features. Simultaneously, line features were extracted from the SLAM dataset using the LSD algorithm. A line segment detector identified line segment features in the image and calculated their length, direction, and position parameters, obtaining a set of line feature vectors representing the structural features of the image.

[0034] Then, joint matching is performed based on the point feature vector set and the line feature vector set. The similarity of point feature descriptors is calculated by Hamming distance, and the similarity of line feature descriptors is calculated by geometric constraints. Then, the random sampling consensus algorithm is used to eliminate mismatched pairs, and multiple point-line joint constraint vectors containing both point feature constraints and line feature constraints are constructed.

[0035] Finally, the on-site reconstruction model is jointly calibrated based on multiple point-line joint constraint vectors. In this process, optimization algorithms such as graph optimization are used to incorporate the point-line joint constraint vectors as constraints into the optimization framework, adjusting the camera pose and map point and line positions in the on-site reconstruction model to minimize the reprojection and geometric errors caused by point and line feature constraints. After this joint calibration process, the second construction monitoring model is finally obtained.

[0036] Step S200: Construct a unified coordinate system, and perform spatial registration and fusion of the first construction monitoring model and the second construction monitoring model according to the unified coordinate system to construct the third construction monitoring model.

[0037] In this embodiment, a unified coordinate system is first constructed, using the WGS84 geographic coordinate system, and a spatial reference framework is established by deploying ground control points and GNSS reference stations. Based on this unified coordinate system, the digital surface model data of the first construction monitoring model and the dense point cloud data of the second construction monitoring model are respectively converted to the unified coordinate framework.

[0038] Then, the first and second construction monitoring models were spatially registered and fused according to a unified coordinate system. In this process, firstly, the common feature points of the two models were extracted using the SIFT feature point matching algorithm, and coordinate transformation parameters were calculated using the least squares method to complete the initial coarse registration. Subsequently, the iterative nearest-point algorithm was used to perform fine registration of the 3D point cloud data of the two models, optimizing the registration accuracy through multiple iterations. In overlapping areas, the high-precision point cloud data of the second construction monitoring model was used to replace the corresponding area data of the first construction monitoring model. In non-overlapping areas, linear interpolation was used for data fusion to ensure terrain continuity. Finally, a seamless 3D model surface was generated using a triangular mesh reconstruction algorithm to construct the third construction monitoring model.

[0039] Step S300: Perform multimodal feature registration constraint analysis based on the first construction monitoring model and the second construction monitoring model to construct the on-site feature registration constraint space.

[0040] Furthermore, in the method provided in the application embodiments, the process of performing multimodal feature registration constraint analysis based on the first construction monitoring model and the second construction monitoring model to construct a field feature registration constraint space further includes:

[0041] A multimodal feature factor is constructed, comprising a first on-site feature factor and a second on-site feature factor. Multi-point feature detection is performed on the first construction monitoring model based on the first on-site feature factor to obtain a first feature vector set for each point. Multi-point feature detection is then performed on the second construction monitoring model based on the second on-site feature factor to obtain a second feature vector set for each point. Point feature fusion is performed based on the first and second feature vector sets for each point to obtain the on-site feature registration constraint space, which includes multiple point feature registration constraint vectors.

[0042] Furthermore, the method provided in the application embodiments also includes:

[0043] The first factor of the site features includes surface cover features, spatial morphology features, topographic elevation features, temporal variation features, and environmental background features. The second factor of the site features includes geometric structure features, spatial topology features, dynamic behavior features, and material texture features.

[0044] In this embodiment, when performing multimodal feature registration constraint analysis based on the first construction monitoring model and the second construction monitoring model, multimodal feature factors are first constructed. These multimodal feature factors include a first site feature factor and a second site feature factor. The first site feature factor encompasses land cover features, spatial morphology features, topographic elevation features, temporal variation features, and environmental background features. Land cover features are extracted from different land cover types in remote sensing images, such as buildings, roads, and vegetation; spatial morphology features describe the shape of buildings or terrain using 3D point cloud data; topographic elevation features are obtained through a digital elevation model (DEM); temporal variation features reflect changes in the construction site at different points in time and are extracted by comparing images from different times; environmental background features are background area information extracted from remote sensing images to describe the overall condition of the surrounding environment.

[0045] The second factor of the site features includes geometric structural features, spatial topological features, dynamic behavior features, and material texture features. Geometric structural features are extracted from point cloud data or structured images to describe the geometric shape of the building; spatial topological features describe the spatial relationships between various elements in the construction site, such as the connection between buildings and roads; dynamic behavior features describe the changes in the construction site, such as construction progress and construction activities; and material texture features are extracted through texture analysis of construction site images to describe the material information of the building surface.

[0046] Next, multi-point feature detection is performed on the first construction monitoring model based on the first factor of on-site characteristics. In this process, land cover features, spatial morphology features, and terrain elevation features are extracted from remote sensing imagery and point cloud data. Corner detection (such as Harris corner detection) and edge detection methods (such as Canny edge detection) are used to identify significant point features in the imagery, and these point features are transformed into point feature vector sets. Each point feature vector contains the spatial location information of the point and a description of local features, such as the direction and texture information of corners. This process yields the first feature vector set for each point.

[0047] Then, based on the second factor of on-site characteristics, multi-point feature detection is performed on the second construction monitoring model. In this process, geometric structural features and spatial topological features are extracted from point cloud data, and plane fitting algorithms such as RANSAC-based algorithms are used to extract the edges and corners of buildings. Furthermore, dynamic behavior features are obtained through analysis of construction progress data or time-series image data, while material texture features are extracted using image texture analysis methods such as gray-level co-occurrence matrix and local binary pattern. Finally, a second feature vector set for each point in the second construction monitoring model is obtained, where each point feature vector contains the geometric, structural, and texture information of that point.

[0048] Then, point feature vector sets obtained from the first and second factors of on-site features are fused. During this process, the similarity between the two sets of feature vectors is calculated using feature matching methods, such as ORB or SIFT descriptor-based matching. Subsequently, metrics such as Hamming distance or Euclidean distance are used to evaluate the similarity of the matched point features. Based on this, geometric constraints, such as planar constraints and spatial position constraints, are used to verify the matching results and eliminate mismatched point pairs. For example, geometric constraints ensure the correct positional relationship of point features in three-dimensional space, optimizing the matching results. Through this process, an on-site feature registration constraint space is obtained. This space includes multiple point feature registration constraint vectors, each containing point feature information from both the first and second construction monitoring models. These constraint vectors, by providing spatial relationships and geometric constraints, ensure that models from two different data sources can be accurately fused in a unified coordinate system.

[0049] Step S400: Perform multimodal feature registration detection and correction on the third construction monitoring model according to the field feature registration constraint space to obtain the fourth construction monitoring model.

[0050] Furthermore, in the method provided in the application embodiments, the method of performing multimodal feature registration detection and correction on the third construction monitoring model according to the site feature registration constraint space to obtain the fourth construction monitoring model further includes:

[0051] Multimodal feature reading is performed based on the third construction monitoring model to obtain multiple feature reading vectors corresponding to multiple site locations. Based on these multiple feature reading vectors, the feature reading vector corresponding to the Kth site location is extracted, where K is a positive integer. Constraint matching is performed on the Kth site location according to the site feature registration constraint space to obtain the Kth site location feature registration constraint vector. A twin comparison evaluation is performed between the Kth site location feature reading vector and the Kth site location feature registration constraint vector to obtain the Kth site location feature twin coefficient. If the Kth site location feature twin coefficient is less than the site feature twin threshold, deviation detection is performed on the Kth site location feature reading vector based on the Kth site location feature registration constraint vector to obtain the Kth site location feature deviation detection vector. Adaptive correction is performed on the third construction monitoring model based on the Kth site location feature deviation detection vector to obtain the fourth construction monitoring model.

[0052] In this embodiment, when performing multimodal feature registration detection and correction on the third construction monitoring model based on the on-site feature registration constraint space, firstly, multimodal feature reading is performed on the third construction monitoring model. By reading the features of each point in the third construction monitoring model, feature data of multiple on-site points are extracted, and corresponding feature reading vectors for multiple points are generated. Each feature reading vector contains information such as the spatial coordinates, geometric features, and local texture features of that point.

[0053] Next, based on the obtained feature reading vectors from multiple points, the feature reading vector corresponding to the Kth point in the field is extracted. The feature reading vector of the Kth point represents the feature data of the Kth point in the third model of construction monitoring, including the spatial coordinates and geometric features of that point.

[0054] Then, constraint matching is performed on the Kth site point based on the site feature registration constraint space. The site feature registration constraint space consists of multiple point feature registration constraint vectors, which define the spatial relationships and geometric features between points in the first and second construction monitoring models. In this process, the Kth site point is directly mapped to the constraint vectors in the site feature registration constraint space, and the K-point feature registration constraint vector corresponding to the Kth site point is extracted.

[0055] Next, a twin comparison evaluation is performed between the feature reading vector at the Kth point and the feature registration constraint vector at the Kth point. Twin comparison involves calculating the similarity between two feature vectors. In this process, the twin coefficient of the Kth point feature is obtained by calculating the Euclidean distance between the feature reading vector at the Kth point and the feature registration constraint vector at the Kth point.

[0056] Subsequently, the twinning coefficient of the Kth point feature is compared with a preset twinning threshold. When the twinning coefficient of the Kth point feature is less than the twinning threshold, deviation detection is performed on the Kth point feature reading vector based on the Kth point feature registration constraint vector. In this process, the difference between the Kth point feature reading vector and the Kth point feature registration constraint vector in each feature dimension is calculated through vector difference operation to generate the Kth point feature deviation detection vector.

[0057] Finally, the third construction monitoring model is adaptively corrected based on the feature deviation detection vector of the Kth point. A least squares optimization algorithm is used to convert each component value of the Kth point feature deviation detection vector into corresponding spatial coordinate corrections and feature parameter corrections. Through coordinate transformation and parameter adjustment, the corrections are applied to the coordinate position and feature parameters of the Kth point in the third construction monitoring model, achieving accurate correction for that point. The above steps are repeated to correct all points in the third construction monitoring model that do not meet the feature twin threshold, ultimately completing the optimization of the entire model and obtaining the fourth construction monitoring model.

[0058] Step S500: Perform multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model to obtain the model unfolding detection anomaly tracing map.

[0059] Furthermore, in the method provided in the application embodiment, the multi-scale unfolding detection anomaly tracing of the fourth construction monitoring model to obtain the model unfolding detection anomaly tracing map further includes:

[0060] Based on the fourth construction monitoring model, macroscopic anomaly detection and tracing are performed to obtain a first unfolded anomaly tracing map; based on the fourth construction monitoring model, mesoscopic anomaly detection and tracing are performed to obtain a second unfolded anomaly tracing map; based on the fourth construction monitoring model, microscopic anomaly detection and tracing are performed to obtain a third unfolded anomaly tracing map; the first unfolded anomaly tracing map, the second unfolded anomaly tracing map, and the third unfolded anomaly tracing map are combined to generate the model unfolded anomaly tracing map.

[0061] In this embodiment, when performing multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model, the macro-scale unfolding detection and anomaly tracing is first performed based on the fourth construction monitoring model. In this process, the fourth construction monitoring model is first unfolded at a macro scale to obtain a first unfolded model image. Then, unfolding detection evaluation factors, including geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy, are activated to evaluate the first unfolded model image, obtaining a first unfolding detection evaluation sequence. Next, anomaly identification is performed on the first unfolding detection evaluation sequence based on the unfolding detection evaluation constraint sequence to obtain first unfolding detection anomaly features. Finally, fault tree tracing is performed on the first unfolding detection anomaly features based on the first unfolded model image to generate a first unfolding detection anomaly tracing map.

[0062] Next, based on the fourth construction monitoring model, mesoscale unfolding and anomaly tracing are performed. First, the model is unfolded at the mesoscale to generate a second unfolded image. Then, unfolding and evaluation factors are activated, and the second unfolded image is evaluated using indicators such as geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy, generating a second unfolding and evaluation sequence. Next, anomaly identification is performed on the second unfolding and evaluation sequence based on the unfolding and evaluation constraint sequence to identify anomalous features. Finally, based on the second unfolded image, fault tree tracing is performed on the anomalous features to generate a second unfolding and anomaly tracing map.

[0063] Similarly, based on the fourth construction monitoring model, micro-scale anomaly tracing is performed. The micro-scale unfolding generates the third unfolded model map, which is then used to activate the unfolded detection evaluation factors for assessment, resulting in the third unfolded detection evaluation sequence. Anomaly identification is then performed based on the unfolded detection evaluation constraint sequence to identify the third unfolded detection anomaly characteristics. Finally, a third unfolded detection anomaly tracing map is generated through fault tree tracing.

[0064] Finally, the first, second, and third unfolded anomaly detection traceability maps are integrated to generate the model unfolded anomaly detection traceability map, which gathers anomaly information at all scales.

[0065] Furthermore, in the method provided in the application embodiment, the macro-scale unfolding detection anomaly tracing based on the fourth construction monitoring model to obtain a first unfolding detection anomaly tracing map further includes:

[0066] The construction monitoring fourth model is expanded at a macro scale to obtain a first expanded model image; the expanded detection evaluation factors are activated, including geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy; the first expanded model image is evaluated based on the expanded detection evaluation factors to obtain a first expanded detection evaluation sequence; anomaly identification is performed on the first expanded detection evaluation sequence based on the expanded detection evaluation constraint sequence to obtain a first expanded detection anomaly feature; fault tree tracing is performed on the first expanded detection anomaly feature based on the first expanded model image to generate a first expanded detection anomaly tracing map.

[0067] In this embodiment, the construction monitoring fourth model is first expanded at a macro scale to generate the first expanded model image. During this process, a large area of ​​the construction site is expanded through spatial transformation and data processing to obtain the spatial relationships and layout of buildings, roads, and other infrastructure. By processing the three-dimensional data of the construction site, these large-scale elements are transformed into visual graphics, resulting in the first expanded model image.

[0068] Next, the unfolded detection evaluation factors are activated, including geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy, and the first unfolded model is evaluated based on these factors. Specifically, in the evaluation of geometric feature accuracy, the shape, size, and location of the buildings are first checked to see if they are consistent with the requirements in the design drawings. In this process, the difference between the geometric features of each building in the first unfolded model and the preset geometric data on the design drawings is calculated to obtain the geometric error. If the difference exceeds the preset accuracy range, the area is considered to have a geometric accuracy deviation. Similarly, texture feature accuracy evaluates the matching of the building surface texture with the actual construction. By comparing the actual texture with a standard texture image, pixel-level differences, such as mean square error (MSE), are calculated to obtain the texture accuracy difference. If the difference is too large, it indicates a problem with texture matching. Next, the boundary feature accuracy is evaluated. This involves calculating the relative position difference between the building boundary and the surrounding environment, for example, using the distance from a point to a line to quantify the boundary alignment between the building and the surrounding environment such as roads or green belts. If the difference exceeds a preset threshold, it is marked as a boundary alignment error. Finally, in evaluating the accuracy of transition features, the smoothness and naturalness of the transition between the building and its surrounding environment are calculated, avoiding abrupt connections. For example, transition accuracy is measured by gradient changes or the smoothness of the transition area, calculating the difference from a standard preset transition area. By accurately evaluating these differences, the evaluation results for each factor are obtained, and a first unfolding detection evaluation sequence is generated based on these results.

[0069] Then, anomaly identification is performed on the first unfolding detection evaluation sequence based on the unfolding detection evaluation constraint sequence. The unfolding detection evaluation constraint sequence includes the accuracy requirements for each evaluation factor, such as the maximum error range of geometric features, the allowable deviation of texture accuracy, and the maximum error of boundary alignment. By comparing the actual evaluation results with these standards, areas that do not meet the standards are identified. If a region deviates from the set allowable range in terms of geometry, texture, boundary, or transition, the region is marked as an anomalous region. All identified anomalous regions are summarized to form the first unfolding detection anomalous feature, indicating the accuracy deviation areas in the first unfolded model diagram.

[0070] Finally, based on the unfolded first image of the model, a fault tree analysis is performed on the anomaly features detected in the first unfolding. In this process, fault tree analysis is conducted by associating each anomaly feature with its possible root causes. For example, if the location of a building deviates from its design location, it may be due to measurement errors or design flaws; if the texture is inconsistent, it may be due to material problems or improper construction. By associating these anomaly features with potential construction problems, design errors, and other causes, a fault tree analysis map of the first unfolded detection is generated.

[0071] Step S600: Based on the model, expand the detection anomaly tracing map and perform adaptive correction on the fourth construction monitoring model to obtain the fifth construction monitoring model.

[0072] In this embodiment, when adaptively correcting the fourth construction monitoring model based on the model unfolding detection anomaly tracing map, the deviation areas in the fourth construction monitoring model are first identified by analyzing the anomaly features marked in the model unfolding detection anomaly tracing map. This map contains detailed information on various anomalies at the construction site, such as geometric deviations of buildings, texture mismatches, and boundary misalignments. Next, using this anomaly information, an adaptive correction algorithm is employed to correct the errors in the fourth construction monitoring model. Specifically, based on the least squares method or graph optimization method, the spatial position, shape, and size of the buildings are adjusted by minimizing the model error to ensure consistency with the design drawings and construction standards. Through gradual adjustments, the fifth construction monitoring model is finally obtained. This model is more accurate in terms of geometry, texture, and boundaries, and can better reflect the actual situation at the construction site.

[0073] In summary, the embodiments of this application have at least the following technical effects:

[0074] This application utilizes remote sensing satellite systems and UAV SLAM systems to monitor construction sites in real time, constructing a first construction monitoring model and a second construction monitoring model. A unified coordinate system is established, and the first and second construction monitoring models are spatially registered and fused according to this unified coordinate system to construct a third construction monitoring model. Multimodal feature registration constraint analysis is performed on the first and second construction monitoring models to construct a site feature registration constraint space. Multimodal feature registration detection and correction are performed on the third construction monitoring model according to the site feature registration constraint space to obtain a fourth construction monitoring model. Multi-scale unfolding detection and anomaly tracing are performed on the fourth construction monitoring model to obtain a model unfolding detection anomaly tracing map. Adaptive correction is performed on the fourth construction monitoring model according to the model unfolding detection anomaly tracing map to obtain a fifth construction monitoring model. This invention solves the technical problem of low accuracy in the fusion of satellite imagery and UAV SLAM data in the prior art. Through multi-source data fusion and precise registration and correction, it achieves the technical effect of improving the accuracy of satellite imagery and UAV SLAM data fusion.

[0075] Example 2, based on the same inventive concept as the method for fusing satellite imagery and UAV SLAM data in the foregoing examples, such as... Figure 2 As shown, this application provides a system for fusing satellite imagery and UAV SLAM data. The system and method embodiments in this application are based on the same inventive concept. The system includes:

[0076] The system comprises the following modules: a real-time monitoring module 11, which monitors the construction site in real time using a remote sensing satellite system and an unmanned aerial vehicle (UAV) SLAM system, and constructs a first construction monitoring model and a second construction monitoring model; a spatial registration and fusion module 12, which constructs a unified coordinate system and performs spatial registration and fusion of the first and second construction monitoring models based on the unified coordinate system, and constructs a third construction monitoring model; a registration constraint analysis module 13, which performs multimodal feature registration constraint analysis on the first and second construction monitoring models, and constructs a site feature registration constraint space; a registration detection and correction module 14, which performs multimodal feature registration detection and correction on the third construction monitoring model based on the site feature registration constraint space, and obtains a fourth construction monitoring model; an anomaly tracing module 15, which performs multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model, and obtains a model unfolding detection anomaly tracing map; and an adaptive correction module 16, which performs adaptive correction on the fourth construction monitoring model based on the model unfolding detection anomaly tracing map, and obtains a fifth construction monitoring model.

[0077] Furthermore, the system is also used to implement the following functions:

[0078] The construction site is monitored in real time by the remote sensing satellite system to obtain satellite image monitoring data; the satellite image monitoring data is cleaned in multiple dimensions according to the satellite image cleaning factor to obtain on-site satellite image data; an on-site satellite image model is constructed based on the on-site satellite image data; the on-site satellite image model is compared and corrected for deviations based on the on-site satellite image data to generate the first construction monitoring model.

[0079] Furthermore, the system is also used to implement the following functions:

[0080] The construction site is monitored in real time by the UAV SLAM system to obtain a SLAM dataset. A site reconstruction model is constructed based on the SLAM dataset. Point features are extracted from the SLAM dataset using the ORB-SLAM algorithm to obtain a set of point feature vectors. Line features are extracted from the SLAM dataset using the LSD algorithm to obtain a set of line feature vectors. Multiple point-line joint constraint vectors are constructed through joint matching of the point and line feature vector sets. The site reconstruction model is then jointly corrected based on these multiple point-line joint constraint vectors to obtain the second construction monitoring model.

[0081] Furthermore, the system is also used to implement the following functions:

[0082] A multimodal feature factor is constructed, comprising a first on-site feature factor and a second on-site feature factor. Multi-point feature detection is performed on the first construction monitoring model based on the first on-site feature factor to obtain a first feature vector set for each point. Multi-point feature detection is then performed on the second construction monitoring model based on the second on-site feature factor to obtain a second feature vector set for each point. Point feature fusion is performed based on the first and second feature vector sets for each point to obtain the on-site feature registration constraint space, which includes multiple point feature registration constraint vectors.

[0083] Furthermore, the system is also used to implement the following functions:

[0084] Multimodal feature reading is performed based on the third construction monitoring model to obtain multiple feature reading vectors corresponding to multiple site locations. Based on these multiple feature reading vectors, the feature reading vector corresponding to the Kth site location is extracted, where K is a positive integer. Constraint matching is performed on the Kth site location according to the site feature registration constraint space to obtain the Kth site location feature registration constraint vector. A twin comparison evaluation is performed between the Kth site location feature reading vector and the Kth site location feature registration constraint vector to obtain the Kth site location feature twin coefficient. If the Kth site location feature twin coefficient is less than the site feature twin threshold, deviation detection is performed on the Kth site location feature reading vector based on the Kth site location feature registration constraint vector to obtain the Kth site location feature deviation detection vector. Adaptive correction is performed on the third construction monitoring model based on the Kth site location feature deviation detection vector to obtain the fourth construction monitoring model.

[0085] Furthermore, the system is also used to implement the following functions:

[0086] Based on the fourth construction monitoring model, macroscopic anomaly detection and tracing are performed to obtain a first unfolded anomaly tracing map; based on the fourth construction monitoring model, mesoscopic anomaly detection and tracing are performed to obtain a second unfolded anomaly tracing map; based on the fourth construction monitoring model, microscopic anomaly detection and tracing are performed to obtain a third unfolded anomaly tracing map; the first unfolded anomaly tracing map, the second unfolded anomaly tracing map, and the third unfolded anomaly tracing map are combined to generate the model unfolded anomaly tracing map.

[0087] Furthermore, the system is also used to implement the following functions:

[0088] The construction monitoring fourth model is expanded at a macro scale to obtain a first expanded model image; the expanded detection evaluation factors are activated, including geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy; the first expanded model image is evaluated based on the expanded detection evaluation factors to obtain a first expanded detection evaluation sequence; anomaly identification is performed on the first expanded detection evaluation sequence based on the expanded detection evaluation constraint sequence to obtain a first expanded detection anomaly feature; fault tree tracing is performed on the first expanded detection anomaly feature based on the first expanded model image to generate a first expanded detection anomaly tracing map.

[0089] Furthermore, the system is also used to implement the following functions:

[0090] The satellite image cleaning factors include geometric correction, atmospheric correction, radiometric correction, image enhancement, and noise suppression.

[0091] Furthermore, the system is also used to implement the following functions:

[0092] The first factor of the site features includes surface cover features, spatial morphology features, topographic elevation features, temporal variation features, and environmental background features. The second factor of the site features includes geometric structure features, spatial topology features, dynamic behavior features, and material texture features.

[0093] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for fusing satellite imagery and unmanned aerial vehicle SLAM data, characterized in that, The method includes: Real-time monitoring of the construction site is conducted using remote sensing satellite systems and UAV SLAM systems to construct a first construction monitoring model and a second construction monitoring model. A unified coordinate system is constructed, and the first construction monitoring model and the second construction monitoring model are spatially registered and fused according to the unified coordinate system to construct a third construction monitoring model; Based on the first construction monitoring model and the second construction monitoring model, multimodal feature registration constraint analysis is performed to construct the on-site feature registration constraint space; Based on the field feature registration constraint space, the third construction monitoring model is subjected to multimodal feature registration detection and correction to obtain the fourth construction monitoring model; The fourth construction monitoring model is expanded at multiple scales to detect and trace anomalies, and an anomaly traceability map of the expanded model is obtained. The construction monitoring fourth model is adaptively corrected based on the model unfolded anomaly tracing map to obtain the construction monitoring fifth model. By analyzing the abnormal features identified in the model unfolded anomaly tracing map, the deviation areas in the construction monitoring fourth model are identified. Using this abnormal information, the error in the construction monitoring fourth model is corrected using an adaptive correction algorithm to obtain the construction monitoring fifth model. Real-time monitoring of the construction site is conducted using remote sensing satellite systems and unmanned aerial vehicle (UAV) SLAM systems, including: The construction site is monitored in real time by the remote sensing satellite system to obtain satellite image monitoring data; The satellite image monitoring data is cleaned in multiple dimensions according to the satellite image cleaning factor to obtain on-site satellite image data. Based on the aforementioned on-site satellite imagery data, an on-site satellite imagery model is constructed; Based on the on-site satellite image data, the on-site satellite image model is compared and corrected for deviations to generate the first construction monitoring model; Real-time monitoring of the construction site is conducted using remote sensing satellite systems and unmanned aerial vehicle (UAV) SLAM systems, including: The construction site is monitored in real time by the UAV SLAM system to obtain a SLAM dataset; Based on the SLAM dataset, construct an on-site reconstruction model; The ORB-SLAM algorithm is used to extract point features from the SLAM dataset to obtain a set of point feature vectors. Line feature extraction is performed on the SLAM dataset based on the LSD algorithm to obtain a set of line feature vectors. Based on the point feature vector set and the line feature vector set, a joint matching is performed to construct multiple point-line joint constraint vectors; The on-site reconstruction model is jointly corrected based on the multiple point-line joint constraint vectors to obtain the second construction monitoring model. Based on the first construction monitoring model and the second construction monitoring model, multimodal feature registration constraint analysis is performed to construct a field feature registration constraint space, including: Construct a multimodal feature factor, which includes a first on-site feature factor and a second on-site feature factor; Based on the first factor of the on-site features, the first construction monitoring model is subjected to multi-point feature detection to obtain the first feature vector set of each point; Based on the second factor of the site features, the second construction monitoring model is subjected to multi-point feature detection to obtain the second feature vector set of each point. Based on the first feature vector set and the second feature vector set of each point, point feature fusion is performed to obtain the field feature registration constraint space, which includes multiple point feature registration constraint vectors. The first factor of the site features includes surface cover features, spatial morphology features, topographic elevation features, temporal variation features, and environmental background features; the second factor of the site features includes geometric structure features, spatial topology features, dynamic behavior features, and material texture features. Based on the site feature registration constraint space, the third construction monitoring model is subjected to multimodal feature registration detection and correction to obtain the fourth construction monitoring model, including: Based on the third construction monitoring model, multimodal feature readings are performed to obtain multiple feature reading vectors corresponding to multiple site locations. Based on the multiple point feature reading vectors, extract the Kth point feature reading vector corresponding to the Kth on-site point, where K is a positive integer; Based on the field feature registration constraint space, the Kth field point is constrained and matched to obtain the Kth point feature registration constraint vector. The twin comparison evaluation is performed between the feature reading vector of the Kth point and the feature registration constraint vector of the Kth point to obtain the twin coefficient of the Kth point feature; If the twinning coefficient of the Kth point feature is less than the twinning threshold of the point feature, the deviation detection of the Kth point feature reading vector is performed on the Kth point feature registration constraint vector to obtain the Kth point feature deviation detection vector; The third construction monitoring model is adaptively corrected based on the feature deviation detection vector of the Kth point to obtain the fourth construction monitoring model; The fourth construction monitoring model is expanded at multiple scales to detect and trace anomalies, resulting in an anomaly traceability map, including: Based on the fourth construction monitoring model, macroscopic scale expansion detection anomaly tracing is performed to obtain the first expansion detection anomaly tracing map. Based on the fourth construction monitoring model, anomaly tracing was performed at the mesoscale to obtain the second anomaly tracing map. Based on the fourth construction monitoring model, micro-scale anomaly tracing is performed to obtain the third unfolded anomaly tracing map. Organize the first unfolded detection anomaly tracing map, the second unfolded detection anomaly tracing map, and the third unfolded detection anomaly tracing map to generate the model unfolded detection anomaly tracing map; Based on the fourth construction monitoring model, macroscopic-scale anomaly detection and tracing are performed to obtain a first unfolded anomaly tracing map, including: Based on the fourth construction monitoring model, a macroscopic scale expansion was performed to obtain the first expanded model diagram. Activate the unfolding detection evaluation factors, which include geometric feature accuracy, texture feature accuracy, boundary feature accuracy, and transition feature accuracy; The first unfolded graph of the model is evaluated based on the unfolding detection evaluation factors to obtain the first unfolding detection evaluation sequence; Anomaly identification is performed on the first unfold detection evaluation sequence based on the unfold detection evaluation constraint sequence to obtain the first unfold detection anomaly features; Based on the model, the first expanded anomaly features are traced using an accident tree to generate the first expanded anomaly tracing map. 2.The method of fusing satellite imagery and UAV SLAM data according to claim 1, wherein, The satellite image cleaning factors include geometric correction, atmospheric correction, radiometric correction, image enhancement, and noise suppression.

3. The system for fusing satellite imagery and unmanned aerial vehicle SLAM data, characterized in that, The system is used to execute the method for fusing satellite imagery and UAV SLAM data as described in any one of claims 1-2, the system comprising: The real-time monitoring module is used to monitor the construction site in real time based on remote sensing satellite systems and UAV SLAM systems, and to build a first construction monitoring model and a second construction monitoring model. The spatial registration and fusion module is used to construct a unified coordinate system and perform spatial registration and fusion of the first construction monitoring model and the second construction monitoring model according to the unified coordinate system to construct the third construction monitoring model. The registration constraint parsing module is used to perform multimodal feature registration constraint parsing based on the first construction monitoring model and the second construction monitoring model, and to construct a field feature registration constraint space. The registration detection and correction module is used to perform multimodal feature registration detection and correction on the third construction monitoring model according to the on-site feature registration constraint space to obtain the fourth construction monitoring model. The anomaly tracing module is used to perform multi-scale unfolding detection and anomaly tracing on the fourth construction monitoring model to obtain an anomaly tracing map of the unfolded model. An adaptive correction module is used to adaptively correct the fourth construction monitoring model based on the anomaly tracing map expanded from the model, thereby obtaining the fifth construction monitoring model.

Images