Building facade point cloud and image spatio-temporal registration method and system

Abstract

The application relates to the technical field of image processing, in particular to a building facade point cloud and image space-time registration method and system, wherein the building facade point cloud and image space-time registration method comprises the following steps: acquiring point cloud data and image data of a building facade; respectively extracting point cloud structure fragments and image boundary fragments in the point cloud data and the image data; matching three-dimensional boundaries of the point cloud structure fragments with the image boundary fragments to establish candidate boundary pairs; acquiring projection residuals between corresponding points in the candidate boundary pairs and space correction qualification states of the projection residuals; and acquiring a space-time registration result. The application sequentially judges the current visual angle boundary support degree, the current window homologous visible range and the same boundary continuous residual correction degree for the candidate boundary pairs, forms the space correction qualification of the projection residuals, makes the correctable residuals enter the registration optimization and the space-time mismatching residuals are isolated, and the error traction of the uncorrectable residuals on the overall registration is avoided.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for spatiotemporal registration of point clouds and images of building facades. Background Technology

[0002] Building facade inspection, surveying, and digital modeling require mapping information on cracks, detachments, water seepage, color differences, and component conditions from camera images to the 3D facade locations represented by point clouds. This achieves precise fusion of texture, defects, and geometric structures. Point cloud data is acquired by ground-based laser scanners, mobile measuring equipment, or UAV LiDAR, recording the 3D geometric boundaries of the wall's main plane, window openings, balcony protrusions, eaves, and pipelines. Image data is acquired by cameras, UAVs, or inspection equipment, recording texture, defects, cracks, stains, and reflections within the current field of view. Registration between point clouds and images is typically optimized based on the camera pose and the overall projection residual between the point cloud projection boundary and the image boundary. By adjusting the camera pose or local projection relationships, the projection residual is minimized, thereby establishing a spatial correspondence between the 3D point cloud and the 2D image.

[0003] Existing registration methods attribute projection residuals to camera pose deviations or projection relationship deviations without distinguishing the sources of these residuals. This results in uncorrectable residuals arising from inconsistencies in spatiotemporal visibility conditions between 3D boundaries and 2D edges being included in spatial optimization. In building facades, elements such as wall main planes, window recesses, balcony protrusions, curtain wall segments, eaves, pipelines, temporary occlusions, and reflective boundaries cause 3D boundaries in the point cloud to have different origins from 2D edges in the image. The point cloud more stably represents depth abrupt changes and geometric boundaries, while the image more clearly shows bright and dark edges, texture lines, and visible contours from the current viewpoint. Stable boundaries of the same structure in the 3D point cloud may lack stable 2D edge support in the current image due to occlusion, compression, or weak textures; shadows, reflections, or temporary occlusion boundaries in the image do not correspond to real 3D structures in the point cloud. These projection residuals caused by inconsistencies in spatiotemporal visibility conditions or object switching, once included in spatial optimization, will influence overall registration, causing defect information in the image to be mapped to incorrect floors, incorrect window rows, or incorrect wall fragments. Summary of the Invention

[0004] This invention provides a method and system for spatiotemporal registration of point clouds and images of building facades to solve the above-mentioned problems.

[0005] The spatiotemporal registration method and system for building facade point cloud and image adopts the following technical solution: In a first aspect, one embodiment of the present invention provides a method for spatiotemporal registration of point cloud and image of building facade. The method includes: acquiring point cloud data and image data of building facade; extracting point cloud structural segments and image boundary segments from the point cloud data and the image data, respectively; matching the three-dimensional boundary of the point cloud structural segment with the image boundary segment to establish candidate boundary pairs; acquiring the projection residual between corresponding points in the candidate boundary pairs and the spatial correction eligibility status of the projection residual; filtering the projection residual based on the spatial correction eligibility status, and performing registration optimization based on the correctable projection residual to obtain spatiotemporal registration results.

[0006] Further, the method for obtaining the spatial correction eligibility status includes: obtaining the current viewpoint boundary support degree based on the image edge support of the three-dimensional boundary in the candidate boundary pair under the current viewpoint; wherein, the current viewpoint boundary support degree is used to quantify the degree to which the three-dimensional boundary obtains two-dimensional edge coverage in the same direction under the current image viewpoint; obtaining the current window's same-source visible range based on the same-source visible status of the three-dimensional boundary in the candidate boundary pair under the current spatiotemporal window; wherein, the current window's same-source visible range is used to quantify the proportion of the three-dimensional boundary that has obtained image edge coverage, which is consistent with the point cloud scanning state and is not occluded in the current field of view; obtaining the same-boundary continuous residual correction degree based on the continuous distribution of the projection residual on the same-source visible boundary of the candidate boundary pair; wherein, the same-boundary continuous residual correction degree is used to quantify the degree to which the projection residual maintains the same direction and continuous amplitude distribution along the same-source visible boundary; and obtaining the spatial correction eligibility status of the projection residual based on the current viewpoint boundary support degree, the current window's same-source visible range, and the same-boundary continuous residual correction degree.

[0007] Further, the step of obtaining the degree of boundary support in the current viewpoint based on the image edge support of the three-dimensional boundary in the candidate boundary pair includes: traversing the ordered sampling pairs in the candidate boundary pair along the boundary arc length; when there is an image edge sampling point in the normal neighborhood of the point cloud projection sampling point, and the edge direction difference between the point cloud projection sampling point and the image edge sampling point does not exceed the direction boundary, marking the sampling point as a supporting sampling point in the same direction; connecting consecutive adjacent supporting sampling points in the same direction along the boundary arc length to obtain the current viewpoint imaging boundary segment; and accumulating the arc length of the point cloud projection boundary point sequence in the candidate boundary pair to obtain the point cloud projection. Total boundary length; for each current viewpoint imaging boundary segment, accumulate the projected arc length between the support sampling points in the same direction within the segment, obtain the overlapping length of the image edges in the same direction, and sum the overlapping lengths of the image edges in the same direction for each segment to obtain the total imaging support length; based on the degree of deviation between the three-dimensional boundary unit extension direction of each current viewpoint imaging boundary segment and the camera's line of sight, obtain the viewpoint compression deviation; based on the viewpoint compression deviation, adjust the overlapping length of the image edges in the same direction for the corresponding imaging boundary segment to obtain the effective support length of the current viewpoint; based on the effective support length of the current viewpoint and the total length of the point cloud projection boundary, obtain the degree of boundary support of the current viewpoint.

[0008] Further, the step of obtaining the same-source visibility range of the current window based on the same-source visibility of the 3D boundary in the current spatiotemporal window based on the candidate boundary alignment includes: for each current viewpoint imaging boundary segment, reading the scanning time period and image capture time, and determining the state change interval based on the component changes, temporary occlusions, and local construction records that overlap with the projection range of the corresponding boundary segment between the scanning time period and the capture time; reading the camera line of sight and the point cloud depth layer, checking whether there is a front structure segment in front of the corresponding boundary segment along the camera line of sight, and marking the overlapping part as a front occlusion interval when the projection range of the front structure segment overlaps with the corresponding boundary segment and the depth of the front structure segment is less than the depth of the corresponding boundary segment; determining the state change length of each state change interval and the front occlusion length of each front occlusion interval; determining the invalid overlap length between the state change interval and the front occlusion interval based on the state change length and the front occlusion length; determining the same-source visibility length of the current window based on the total imaging support length and the invalid overlap length; and obtaining the same-source visibility range of the current window based on the same-source visibility length of the current window and the total imaging support length.

[0009] Further, the step of obtaining the degree of continuous residual correction based on the continuous distribution of the projection residuals on the same visible boundary of the candidate boundary pair includes: reading the projection residual vector for the residual sampling points within the same visible length of the current window, and projecting the projection residual vector onto the local correction direction; reading the projection sign and residual amplitude of each residual sampling point sequentially along the boundary arc length; connecting adjacent residual sampling points with the same projection sign and residual amplitude changes not exceeding the jump boundary to obtain correctable residual segments, and accumulating the boundary arc length of each correctable residual segment to obtain the total length of correctable residual segments; counting the total number of adjacent residual relationships on the same visible boundary of the current window, marking adjacent residual relationships with reversed projection signs or residual amplitude changes exceeding the jump boundary as residual jump relationships, and accumulating the residual jump count to obtain the residual jump ratio; and obtaining the degree of continuous residual correction based on the total length of correctable residual segments, the same visible length of the current window, and the residual jump ratio.

[0010] Further, obtaining the spatial correction eligibility status of the projection residual based on the current viewpoint boundary support level, the current window's homologous visible range, and the homologous boundary continuous residual correction level includes: based on the comparison result of the current viewpoint boundary support level and the viewpoint boundary for each candidate boundary pair, determining the spatial correction eligibility status of candidate boundary pairs whose current viewpoint boundary support level is lower than the viewpoint boundary as not having spatial correction eligibility status; for candidate boundary pairs whose current viewpoint boundary support level reaches the viewpoint boundary, based on the comparison result of the current window's homologous visible range and the spatiotemporal boundary, determining the spatial correction eligibility status of candidate boundary pairs whose current window's homologous visible range is lower than the spatiotemporal boundary as spatiotemporal mismatch status; for candidate boundary pairs whose current window's homologous visible range reaches the spatiotemporal boundary, based on the comparison result of the homologous boundary continuous residual correction level and the correction boundary, determining the spatial correction eligibility status of candidate boundary pairs whose homologous boundary continuous residual correction level reaches the correction boundary as having spatial correction eligibility status, and determining the spatial correction eligibility status of candidate boundary pairs whose homologous boundary continuous residual correction level does not reach the correction boundary as not having spatial correction eligibility status.

[0011] Further, the step of filtering the projection residuals based on the spatial correction eligibility status and performing registration optimization based on the correctable projection residuals to obtain spatiotemporal registration results includes: adding the projection residuals of candidate boundary pairs with a spatial correction eligibility status to the spatial registration optimization set; calculating the camera pose correction or local projection correction based on the projection residuals in the spatial registration optimization set and updating the registration relationship between the image and the point cloud; isolating the projection residuals of candidate boundary pairs with a spatial correction eligibility status of spatiotemporal mismatch or without a spatial correction eligibility status as uncorrectable projection residuals, and not participating in global spatial optimization; outputting occlusion verification, resampling, or time window adjustment markers for structural segments with a spatial correction eligibility status of spatiotemporal mismatch; and outputting the spatial correction eligibility status and registration status of each structural segment.

[0012] Further, obtaining the projection residual between corresponding points in the candidate boundary pair includes: calculating the cumulative arc length for the point cloud projection boundary point sequence and the image boundary point sequence in the candidate boundary pair respectively, and normalizing their respective cumulative arc lengths; interpolating points on the point cloud projection boundary point sequence and the image boundary point sequence according to the same normalized arc length position to form an ordered sampling pair; wherein, the ordered sampling pair includes point cloud projection boundary sampling points and image boundary sampling points at the same normalized arc length position; and obtaining the projection residual based on the two-dimensional offset vector between the point cloud projection boundary sampling points and the image boundary sampling points.

[0013] Further, the step of extracting point cloud structure fragments and image boundary fragments from the point cloud data and the image data respectively includes: dividing the point cloud data into structure fragments based on similar normals and similar distances from points to local planes; extracting three-dimensional boundary points at abrupt changes in depth, curvature, or outer contours of adjacent fragments; arranging the three-dimensional boundary points along the boundary extension direction to obtain point cloud structure fragments; extracting two-dimensional edge points from the image data; merging edge segments that are continuous in direction and have close endpoint distances; arranging the merged edge segments in order of arc length from endpoint to endpoint to obtain image boundary fragments. The step of matching the three-dimensional boundary of the point cloud structure segment with the image boundary segment to establish candidate boundary pairs includes: projecting the three-dimensional boundary point array of the point cloud structure segment onto the image plane; for each image boundary segment, selecting point cloud projection boundary segments with similar directions and similar endpoint order within the adjacent projection range to establish candidate boundary pairs.

[0014] Secondly, another embodiment of the present invention provides a spatiotemporal registration system for point clouds and images of building facades, including: The data acquisition module is used to acquire point cloud data and image data of the building facade; The fragment extraction module is used to extract point cloud structure fragments and image boundary fragments from the point cloud data and the image data, respectively. The boundary matching module is used to match the three-dimensional boundary of the point cloud structure segment with the image boundary segment to establish candidate boundary pairs; The residual and qualification acquisition module is used to acquire the projection residual between corresponding points in the candidate boundary pair and the spatial correction qualification status of the projection residual. The screening and registration module is used to screen the projection residuals based on the spatial correction eligibility status, and to perform registration optimization based on the correctable projection residuals to obtain the spatiotemporal registration result.

[0015] The beneficial effects of the technical solution of the present invention are: In this embodiment of the invention, point cloud data and image data of the building facade are acquired; point cloud structure fragments and image boundary fragments are extracted from the point cloud data and image data, respectively; the three-dimensional boundaries of the point cloud structure fragments are matched with the image boundary fragments to establish candidate boundary pairs; the projection residuals between corresponding points in the candidate boundary pairs and the spatial correction eligibility status of the projection residuals are acquired; the projection residuals are filtered based on the spatial correction eligibility status, and registration optimization is performed based on the correctable projection residuals to obtain the spatiotemporal registration results.

[0016] This invention, by sequentially judging the current viewpoint boundary support level, the current window's same-source visible range, and the degree of correction of continuous residuals along the same boundary for candidate boundary pairs, forms the spatial correction qualification of projection residuals. This allows correctable residuals to enter the registration optimization while isolating spatiotemporally mismatched residuals, avoiding the erroneous pull of uncorrectable residuals on the overall registration. On the other hand, it reduces registration deviations caused by window openings, balcony protrusions, eaves, pipelines, scaffolding obstructions, and repeated curtain wall segmentation, ensuring that the registration results of point clouds and images conform to the true spatiotemporal relationship between the current image field of view and the point cloud scanning segment. Furthermore, it transforms projection residuals that do not have spatial correction qualification into occlusion verification, re-sampling, or time window adjustment markers, providing a directional basis for subsequent local data acquisition and temporal verification, avoiding information loss caused by directly discarding uncorrectable residuals. Moreover, it improves the accuracy of mapping image textures, inspection marks, and appearance changes to point cloud structural segments, ensuring that information on defects such as cracks, stains, and peeling can be accurately mapped to the true location of the three-dimensional facade. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for spatiotemporal registration of point clouds and images of building facades provided in an embodiment of the present invention. Figure 2 A flowchart illustrating the spatial correction qualification status determination scheme provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the architecture of the building facade point cloud and image spatiotemporal registration system provided in an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following detailed description, in conjunction with the accompanying drawings and preferred embodiments, provides a detailed account of the specific implementation methods, structures, features, and effects of the method proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments may be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0020] The following description, in conjunction with the accompanying drawings, details the specific scheme of the method and system for spatiotemporal registration of point clouds and images of building facades provided by this invention.

[0021] like Figure 1 As shown, this embodiment of the invention provides a method for spatiotemporal registration of point clouds and images of building facades, including: Step S110: Obtain point cloud data and image data of the building facade.

[0022] The aforementioned point cloud data can be acquired through ground-based laser scanners, mobile measuring devices, or UAV LiDAR, primarily recording the three-dimensional geometric information of the building facade. The point cloud data mainly includes three-dimensional coordinates, normals, point density, scanning time period, scan segment number, and scanning perspective. The three-dimensional coordinates are used to characterize the spatial location of structures such as the main plane of the wall, window openings, balcony protrusions, eaves, and pipelines. The normals and point density are used for subsequent structural segment division and boundary extraction. The scanning time period, scan segment number, and scanning perspective are used to trace the temporal and spatial origin of the point cloud acquisition.

[0023] The aforementioned image data can be acquired via cameras, drones, or inspection equipment at times and from perspectives different from those used in point cloud acquisition. It primarily records two-dimensional visual information within the current field of view. The image data mainly includes image frames, capture time, camera position, field of view orientation, two-dimensional edges, and occlusion regions. Image frames carry apparent information such as texture, defects, cracks, stains, reflections, and occlusion fronts. Capture time, camera position, and field of view orientation are used to establish the spatiotemporal correspondence between the image and the point cloud. Two-dimensional edges and occlusion regions provide visual evidence for subsequent matching with three-dimensional boundaries and for determining the source of residuals.

[0024] After acquiring point cloud data, it can be unified to the architectural coordinate system. This ensures that multiple segments of point cloud data from different scanning devices and time periods have a unified spatial reference. This unification process transforms the local coordinate system of each scanning segment to the architectural coordinate system through coordinate transformation, ensuring that the three-dimensional positional relationships of the main plane of the wall, window openings, balconies, and eaves are consistently expressed in the global space. After acquiring image data, projection relationships can be established based on the initial camera pose or coarse registration results. This maps the three-dimensional point cloud coordinates in the architectural coordinate system to the image plane, forming a geometric transformation channel between three-dimensional space and two-dimensional image. This provides a foundation for subsequently establishing candidate boundary pairs and calculating projection residuals.

[0025] Point cloud data and image data are organized under the same building facade object to maintain the object-level association between 3D structural information and 2D visual information. The point cloud side retains the 3D positional relationships of walls, window openings, balconies, and eaves to maintain the depth and curvature abrupt changes of adjacent segments during structural segment division; the image side retains the 2D edges and occlusion areas actually seen in the current field of view to determine whether the projection residual comes from spatial deviation or from the current image not seeing the same structural boundary.

[0026] Step S120: Extract point cloud structure fragments and image boundary fragments from the point cloud data and image data, respectively.

[0027] The aforementioned point cloud structure fragments are geometric boundary carriers formed by dividing the point cloud data of building facades into local structural units such as the main plane of the wall, the sidewalls of window openings, the front edge of balconies, eaves, and pipelines according to geometric features, and then extracting three-dimensional boundary points at depth abrupt changes, curvature abrupt changes, or the outer contour of adjacent fragments and arranging them along the boundary extension direction. They mainly record the fragment number, the three-dimensional boundary point sequence, adjacent fragments, and the scanning time period, used to stably represent depth abrupt changes and geometric boundaries in three-dimensional space. The aforementioned image boundary fragments are visual boundary carriers formed by merging two-dimensional edge points in an image into line segments according to directional continuity and endpoint proximity, and then arranging them in the arc length order from endpoint to endpoint. They record the boundary number, two-dimensional point sequence, boundary direction, endpoint order, and the image region where they are located, used to represent bright and dark edges, texture lines, and visible contours in a two-dimensional image from the current viewpoint. The purpose of extracting point cloud structural fragments and image boundary fragments is to establish a correspondence between the geometric boundaries in the 3D point cloud and the visual boundaries in the 2D image, so that the same facade structure forms a matching boundary object in both data modalities. This provides a structured processing unit for subsequent establishment of candidate boundary pairs and calculation of projection residuals. The following describes one optional extraction scheme for point cloud structural fragments and image boundary fragments: Optionally, step S120 includes: dividing the point cloud data into structural segments based on similar normals and similar distances from points to local planes; extracting three-dimensional boundary points at abrupt changes in depth, curvature, or outer contours of adjacent segments; arranging the three-dimensional boundary points along the boundary extension direction to obtain point cloud structural segments; extracting two-dimensional edge points from the image data; merging edge segments that are continuous in direction and have similar endpoint distances; arranging the merged edge segments in order of arc length from endpoint to endpoint to obtain image boundary segments.

[0028] When dividing point cloud structural segments, two geometric criteria can be used: similarity of normal directions and similarity of distance from points to local planes. These criteria organize the point cloud data of building facades into local geometric units such as the main plane of the wall, the sidewalls of window openings, the front edge of balconies, eaves, and pipelines. The similarity of normal directions ensures that the normal directions of the point cloud within the same structural segment are consistent, guaranteeing that the segment corresponds to a local plane or regular curved surface. The similarity of distance from points to local planes concentrates the point cloud distribution within the segment near the local plane, excluding noise points far from the local plane or points belonging to other structures. These two criteria work together to aggregate continuous and geometrically consistent 3D point clouds into structural segments with clear physical meaning, making walls, window openings, balconies, and eaves independently processable geometric objects in 3D space.

[0029] The aforementioned depth abrupt changes correspond to depth discontinuities between adjacent segments, such as the depth step between the sidewall of a window opening and the main plane of the wall; the aforementioned curvature abrupt changes correspond to locations on the same surface where the geometry changes significantly, such as the edge where the front edge of a balcony meets the wall; the aforementioned segment outlines correspond to the outer perimeter boundaries of structural segments within the overall facade. The three-dimensional points at depth abrupt changes, curvature abrupt changes, and segment outlines exhibit clear geometric discontinuities and can stably represent the structural boundaries of the building facade; therefore, they are extracted as three-dimensional boundary points.

[0030] The extracted 3D boundary points are arranged along the boundary extension direction to form point cloud structure fragments. The arrangement order follows the natural extension trajectory of the boundary in space, ensuring geometric continuity of the 3D boundary point sequence. Each point cloud structure fragment records the fragment number, the 3D boundary point sequence, adjacent fragments, and the scanning time period. The fragment number is used to uniquely identify the structural unit, the 3D boundary point sequence is used for subsequent projection onto the image plane, the adjacent fragment information is used to maintain the topological relationship between structures, and the scanning time period is used to trace the point cloud acquisition time window corresponding to the fragment.

[0031] The aforementioned two-dimensional edge points can be extracted from image frames using edge detection operators or visual models. These points correspond to locations in the image where there are significant changes in brightness, including texture lines, visible contours, and apparent boundary changes. These two-dimensional edge points constitute the initial samples of structural boundaries in the image, but the initial extraction results are usually discrete points or short, fragmented line segments, and have not yet formed boundary objects with complete geometric meaning. It should be noted that the aforementioned edge detection operators and visual models can be implemented using mature existing technologies. For their implementation methods and working principles, please refer to relevant technologies; the embodiments of this invention will not elaborate further.

[0032] Edge segments with continuous direction and close endpoint distances are merged into longer boundary segments. The direction continuity constraint ensures that adjacent segments tend to have a consistent orientation, maintaining a smooth geometric direction in the merged segment. The endpoint distance constraint ensures that adjacent segments connect spatially, avoiding incorrect connection of distant independent segments. This merging operation organizes discrete, short, fragmented edges into continuous boundary segments, reducing edge breaks caused by local noise or weak textures, and creating a complete and coherent geometric representation of structural boundaries in the image on the two-dimensional plane.

[0033] The merged edge segments are arranged in arc length order from endpoint to endpoint, forming image boundary segments. The arc length order unfolds sequentially from the boundary start point to the endpoint, ensuring that the 2D point array remains ordered along the boundary extension direction. Each image boundary segment records the boundary number, 2D point array, boundary direction, endpoint order, and the image region it belongs to. The boundary number uniquely identifies the visual boundary, the 2D point array is used for subsequent matching with the point cloud projection boundary points, the boundary direction is used to determine consistency with the 3D boundary projection direction, the endpoint order is used to establish the arc length correspondence with the point cloud boundary point array, and the image region it belongs to is used to locate the spatial position of the boundary in the image frame. Image boundary segments structurally organize the 2D visual boundary from discrete edge points, providing a 2D processing unit for subsequent matching with point cloud structure segments.

[0034] It should be noted that the acquisition of the aforementioned point cloud structural segments is not limited to the partitioning methods based on normals and local planar distances. Point cloud boundaries can also be formed by planar segmentation, curvature abrupt change detection, depth fracture recognition, or semantic segmentation results, as long as the geometric boundaries of the building facade can be stably extracted in three-dimensional space. Of course, the acquisition of image boundary segments is not limited to the aforementioned edge detection and line segment merging methods. Image boundaries can also be formed by traditional edge detection, line segment detection, or visual model candidate boundaries, as long as the visible contours under the current viewpoint can be represented in the two-dimensional image.

[0035] Step S130: Match the 3D boundary of the point cloud structure fragment with the image boundary fragment to establish candidate boundary pairs.

[0036] The aforementioned candidate boundary pairs are boundary combination objects established after the 3D boundaries of point cloud structural segments and image boundary segments form a proximity correspondence in position, orientation, and endpoint order. They can characterize the spatial proximity of 3D geometric boundaries and 2D visual boundaries. The purpose of establishing candidate boundary pairs in this embodiment of the invention is to establish a preliminary correspondence between the geometric boundaries in the 3D point cloud modality and the visual boundaries in the 2D image modality, so that the 3D structural information and 2D appearance information, originally in different data spaces, form a directly comparable and quantifiable correlation carrier, providing processing objects for subsequent calculation of projection residuals and determination of spatial correction eligibility based on the source of residuals. An optional scheme for establishing candidate boundary pairs is described below: Optionally, step S130 includes: projecting the three-dimensional boundary point array of the point cloud structure fragment onto the image plane; for each image boundary fragment, selecting point cloud projection boundary fragments with similar directions and similar endpoint order within the adjacent projection range to establish candidate boundary pairs.

[0037] Projecting the 3D boundary point sequence of a point cloud structure fragment onto the image plane is the process of converting 3D coordinates in the architectural coordinate system into 2D coordinates in the image plane through projection relationships. The projection relationship is established based on camera intrinsic parameters and pose parameters. The camera intrinsic parameters determine the perspective transformation relationship from 3D space to the image plane, while the pose parameters determine the camera's spatial position and orientation in the architectural coordinate system. This projection relationship can be the initial camera calibration result or the preliminary registration parameters obtained during the coarse registration stage. Each point in the 3D boundary point sequence forms a projection point on the image plane after projection. The entire 3D boundary point sequence, after projection, forms a point cloud projection boundary fragment, ensuring that the 3D geometric boundary and the 2D image boundary are in the same data space.

[0038] The projected point cloud boundary segments and image boundary segments form a spatial distribution on the image plane. For each image boundary segment, point cloud boundary segments are selected within a neighboring projection range. This neighboring range is determined based on the spatial distance on the image plane, and only spatially close boundaries are considered for matching, avoiding incorrectly establishing correspondences between independent boundaries that are far apart. This spatial proximity constraint limits the establishment of candidate boundary pairs to a locally geometrically consistent region, reducing long-distance mismatches caused by repetitive structures or similar textures on building facades.

[0039] The direction similarity constraint requires that the boundary direction of the point cloud projection boundary segment and the boundary direction of the image boundary segment be consistent in the image plane. This constraint is achieved by comparing the extension directions of the two boundaries, ensuring that the projection direction of the 3D boundary in the current viewpoint matches the actual visual boundary direction observed in the image, and eliminating incorrect pairings of boundaries with opposite or nearly perpendicular directions. The endpoint order similarity constraint requires that the start and end order of the point cloud projection boundary segment and the start and end order of the image boundary segment be consistent in the boundary extension direction, so that the two boundaries form a matching in the same direction in the endpoint correspondence, providing a correct endpoint correspondence basis for subsequently establishing ordered sampling pairs along the boundary arc length.

[0040] Point cloud projection boundary segments and image boundary segments that meet the three conditions of proximity in projection range, similar orientation, and similar endpoint order are combined into candidate boundary pairs. It is understood that candidate boundary pairs only represent the positional proximity of the 3D geometric boundary and the 2D visual boundary on the image plane, and are not directly equivalent to registration boundaries that can be used for camera pose correction. When the vertical edges of window openings, curtain wall grid lines, and pipeline edges are close in distance in the 2D image, even if candidate boundary pairs are formed, the projection residuals on them may still originate from differences in spatiotemporal visibility conditions or object switching, and cannot be directly used as a basis for spatial correction. The establishment of candidate boundary pairs provides a directly operable object carrier for subsequent calculations of ordered sampling pairs and projection residuals, giving residual calculation and spatial correction eligibility judgment a clear boundary correspondence basis.

[0041] The camera pose parameters upon which the above projection relationships are based can be obtained through a global positioning system, inertial measurement unit, or preliminary feature matching, and are not limited to a specific coarse registration algorithm. Furthermore, the spatial distance threshold for adjacent projection ranges, the angle threshold for similar directions, and the endpoint offset threshold for similar endpoint sequences can be adaptively adjusted based on the structural complexity of the building facade, point cloud density, and image resolution. For areas with repetitive curtain wall grids or dense pipelines, the adjacent range threshold can be appropriately tightened while the directional tolerance can be relaxed to ensure the recall rate of candidate boundary pairs while maintaining matching accuracy.

[0042] Step S140: Obtain the projection residuals between corresponding points in the candidate boundary pair and the spatial correction eligibility status of the projection residuals.

[0043] The aforementioned projection residual is a two-dimensional offset vector between the point cloud projection boundary sampling point and the image boundary sampling point at the same normalized arc length position in the candidate boundary pair. It quantifies the degree of spatial deviation between the three-dimensional geometric boundary and the two-dimensional visual boundary under the current projection relationship. The purpose of obtaining the projection residual is to provide a quantifiable deviation basis for subsequent judgment of the source of the residual, transforming the spatial deviation of the candidate boundary pair from an abstract geometric non-coincidence into a concrete numerical expression, thereby supporting the determination of whether the projection residual is qualified for spatial correction, in order to distinguish between correctable residuals caused by camera pose or local projection deviations and uncorrectable residuals caused by inconsistencies in spatiotemporal visibility conditions or object switching. The following introduces an optional scheme for obtaining the aforementioned projection residual: Optionally, the above-mentioned method of obtaining the projection residual between corresponding points in the candidate boundary pair includes: calculating the cumulative arc length for the point cloud projection boundary point sequence and the image boundary point sequence in the candidate boundary pair respectively, and normalizing the respective cumulative arc length; interpolating points on the point cloud projection boundary point sequence and the image boundary point sequence according to the same normalized arc length position to form an ordered sampling pair; wherein, the ordered sampling pair includes point cloud projection boundary sampling points and image boundary sampling points at the same normalized arc length position; and obtaining the projection residual based on the two-dimensional offset vector between the point cloud projection boundary sampling points and the image boundary sampling points.

[0044] The cumulative arc length is calculated for both the point cloud projection boundary point sequence and the image boundary point sequence in the candidate boundary pairs. Starting from the boundary origin, the cumulative arc length is calculated by progressively adding the Euclidean distance between adjacent points along the boundary extension direction, forming an arc length sequence characterizing the geometric length of the boundary. This calculation transforms discrete 3D and 2D boundary points into boundary curves with continuous length measurements, providing a unified length benchmark for establishing a one-to-one correspondence across modalities.

[0045] The cumulative arc lengths of each sample location are normalized to the range of 0 to 1, with the total arc length of the entire boundary as the denominator. This normalization maps the cumulative arc length at each sampling location to a relative proportional value. This process resolves the inconsistency in the number of sampling points between the point cloud projection boundary and the image boundary due to differences in the density of the original point array or differences in physical length, making the two boundaries comparable in dimensionless relative arc length coordinates. Points are interpolated on the point arrays of the point cloud projection boundary and the image boundary according to the same normalized arc length position, forming ordered sampling pairs. The interpolation operation searches for points corresponding to the positions in the normalized arc length coordinate system. When there is no point in the original point array that is exactly located at the target normalized arc length position, a sampling point at that position is generated through linear interpolation of adjacent points. The sampling order follows the cumulative arc length order from the boundary start point to the end point, ensuring that the ordered sampling pairs maintain a consistent arrangement order in the boundary extension direction. Each ordered sampling pair consists of a point cloud projection boundary sampling point and an image boundary sampling point at the same normalized arc length position, forming a one-to-one correspondence in the geometric position of the boundary. The projection residual is obtained based on the two-dimensional offset vector between the point cloud projection boundary sampling points and the image boundary sampling points. The two-dimensional offset vector is calculated in the image plane by subtracting the two-dimensional coordinates of the point cloud projection boundary sampling points from the two-dimensional coordinates of the image boundary sampling points. Its direction and magnitude characterize the spatial deviation of the three-dimensional geometric boundary relative to the two-dimensional visual boundary under the current projection relationship. This projection residual quantifies the degree of spatial non-coincidence between corresponding points on the candidate boundary pair, providing a numerical basis for subsequently determining whether the source of the residual qualifies for spatial correction.

[0046] It should be noted that, in addition to the normalized arc length interpolation scheme mentioned above, the projection residual can also be determined by the following schemes: (1) Determination scheme based on nearest neighbor matching: Perform Euclidean distance search between each point in the point cloud projection boundary point column and the image boundary point column in the image plane, and determine the closest image boundary point as the corresponding point. The two-dimensional offset vector between the point cloud projection point and the corresponding image boundary point is the projection residual. (2) Determination scheme based on the distance from point to line segment: Treat the image boundary segment as a line segment composed of continuous two-dimensional edge points, calculate the shortest perpendicular distance from each point cloud projection boundary point to the line segment, and use the perpendicular distance vector as the projection residual. (3) Determination scheme based on feature similarity: Extract the local gradient direction or curvature features of the point cloud projection boundary points and the image boundary points respectively, establish the correspondence through the similarity measure of feature descriptors, and then calculate the two-dimensional offset vector between the corresponding points as the projection residual.

[0047] like Figure 2 As shown, optionally, the method for obtaining the above-mentioned spatial correction eligibility status includes: Step S141: Based on the image edge support of the 3D boundary in the candidate boundary pair under the current viewpoint, obtain the boundary support degree of the current viewpoint; wherein, the boundary support degree of the current viewpoint is used to quantify the degree to which the 3D boundary obtains 2D edge coverage in the same direction under the current image viewpoint.

[0048] The aforementioned boundary support level from the current viewpoint is used to determine whether the 3D boundaries in the point cloud have achieved stable and oriented 2D edge coverage under the current image viewpoint. This is the structural comparability basis for judging the eligibility of candidate boundaries for subsequent spatial correction. In building facades, the sides of balconies, window openings, and eaves exhibit significant depth abrupt changes in the point cloud, but these are compressed into shorter sides in oblique views. Areas near weakly textured walls lack stable image edges, and reflections from glass curtain walls create bright and dark lines unrelated to the actual structure. In these cases, the 3D boundaries lack effective 2D edge support in the current image. Without obtaining the boundary support level from the current viewpoint, it is impossible to distinguish whether the 3D boundaries have formed comparable effective visual boundaries under the current viewpoint. This leads to candidate boundary pairs lacking stable image edge support directly participating in subsequent calculations, causing their projection residuals to be mistakenly treated as spatial registration errors. The purpose of obtaining the boundary support level of the current viewpoint in this embodiment of the invention is to exclude candidate boundary pairs lacking 2D edge coverage in the same direction from subsequent spatial correction eligibility judgment, prevent boundary residuals that cannot be stably imaged under the current viewpoint from entering registration optimization, and ensure that only 3D boundaries with sufficient image edge support have a structurally comparable basis and enter the next level of screening. The following describes an optional scheme for determining the boundary support level of the current viewpoint: Optionally, step S141 includes: traversing the ordered sampling pairs in the candidate boundary pairs along the boundary arc length; when there is an image edge sampling point in the normal neighborhood of the point cloud projection sampling point, and the edge direction difference between the point cloud projection sampling point and the image edge sampling point does not exceed the direction boundary, marking the sampling point as a support sampling point in the same direction; connecting consecutive adjacent support sampling points in the same direction along the boundary arc length to obtain the current viewpoint imaging boundary segment; accumulating the arc length of the point cloud projection boundary point column in the candidate boundary pairs to obtain the total length of the point cloud projection boundary; for each current viewpoint imaging boundary segment, accumulating the projection arc length between support sampling points in the same direction within the segment to obtain the overlapping length of the image edges in the same direction, and accumulating the overlapping lengths of the image edges in the same direction of each segment to obtain the total length of imaging support; obtaining the viewpoint compression deviation based on the degree of deviation between the three-dimensional boundary unit extension direction of each current viewpoint imaging boundary segment and the camera viewing direction; adjusting the overlapping length of the image edges in the same direction of the corresponding imaging boundary segment based on the viewpoint compression deviation to obtain the effective support length of the current viewpoint; and obtaining the degree of support of the current viewpoint boundary based on the effective support length of the current viewpoint and the total length of the point cloud projection boundary.

[0049] The ordered sampling pairs in the candidate boundary pairs are traversed sequentially along the boundary arc length. Each sampling pair is checked to see if it satisfies the same-direction support condition, which includes two constraints: an image edge sampling point exists within the normal neighborhood of the point cloud projection sampling point, and the edge direction difference between the point cloud projection sampling point and the image edge sampling point does not exceed the direction boundary. The normal neighborhood constraint ensures that the image edge sampling point is located within the local normal range of the point cloud projection sampling point, making the 2D edge geometrically close to the projected position of the 3D boundary; the direction boundary constraint ensures that the projection direction of the 3D boundary tends to be consistent with the direction of the image edge, making the 2D edge geometrically support the 3D boundary. Sampling points that simultaneously satisfy both constraints are marked as same-direction support sampling points, indicating that the 3D boundary at that location has achieved effective 2D edge coverage under the current image viewpoint.

[0050] The aforementioned consecutive adjacent support sampling points in the same direction along the boundary arc length are connected to form the current viewpoint imaging boundary segment. The connection operation aggregates discretely distributed support sampling points in the same direction into continuous geometric segments along the arc length dimension, thus extending the support of the image edge to the three-dimensional boundary from isolated points to boundary segments with length. A candidate boundary pair forms Each current viewpoint imaging boundary segment is independent of the others in terms of boundary arc length, and corresponds to the image edge coverage obtained in different local areas on the three-dimensional boundary.

[0051] The total length of the point cloud projection boundary is obtained by accumulating the arc lengths of the candidate boundary points aligned with the projection boundary point sequence. This length characterizes the complete projection range of the 3D boundary on the image plane and serves as the denominator for measuring the image edge coverage ratio. For each current viewpoint imaging boundary segment, the projected arc lengths between the supporting sampling points in the same direction within the segment are accumulated to obtain the overlapping length of the image edges in the same direction. Length of overlap between image edges in the same direction Able to quantify the first The extent to which the image edges and the 3D boundary projections coincide geometrically within each imaging boundary segment. The total length of the imaging support is obtained by summing the coincidence lengths of the image edges in the same direction for each segment. This length represents the total coverage of the 3D boundary by all imaging boundary segments from the current image viewpoint.

[0052] The 3D boundary unit extension direction of each current viewpoint imaging boundary segment and the camera's line of sight are read, and the absolute value of the cosine of the angle between them is calculated as the viewpoint compression deviation. The extension direction of the 3D boundary unit represents the orientation of the boundary in 3D space, while the camera's line-of-sight direction represents the observation direction from the camera's optical center to the boundary region. When the orientation of the 3D boundary tends to be parallel to the camera's line-of-sight direction, the absolute value of the cosine of the included angle approaches 1, and the angle compression deviation increases, indicating that the 3D boundary is severely compressed under the current oblique viewing angle, and the support contribution of the image edge to the boundary should be weakened. When the orientation of the 3D boundary tends to be perpendicular to the camera's line-of-sight direction, the absolute value of the cosine of the included angle approaches 0, and the angle compression deviation decreases, indicating that the 3D boundary is fully unfolded under the current viewing angle, and the support contribution of the image edge should be preserved. The overlap length of image edges in the same direction is adjusted by subtracting the angle compression deviation from 1 to obtain the effective support length for the current viewing angle. The above adjustment can reduce the coverage contribution of the boundary segment with severe oblique compression, avoiding the excessive inclusion of the compressed short side in the support range.

[0053] Current perspective boundary support level Obtained using the following formula:

[0054] In the formula, For the first The degree of support of the current perspective boundary for each candidate boundary pair. For dimensionless parameters, The larger the value of , the more complete the two-dimensional edge coverage in the same direction is obtained from the current viewpoint; This represents the number of image boundary segments at the current viewpoint. For the first The length of overlap between the image edges in the same direction of each imaging boundary segment; To compress the deviation of the viewpoint, This is a dimensionless parameter, with a value range from 0 to 1; The total length of the point cloud projection boundary. and The units are the same. The numerator is the sum of the effective support lengths of each imaging boundary segment after adjustment for viewpoint compression deviation, and the denominator is the total length of the point cloud projection boundary. This is the normalization operation function. The degree of boundary support at the current viewpoint. By measuring the ratio between the effective support length and the total length of the projected boundary, the degree to which the 3D boundary achieves sufficient coverage of the 2D edges in the same direction under the current image viewpoint is quantified. This current viewpoint boundary support level... As a structurally comparable basis, candidate boundary pairs with low support lack stable image edge coverage, and these candidate boundary pairs are not included in the calculation of the homologous visible range of the subsequent current window.

[0055] The acquisition of the current viewpoint boundary support is not limited to the above-mentioned arc length accumulation and viewpoint compression adjustment schemes. It can also be achieved by: (1) Scheme based on the boundary overlap length ratio: directly calculate the ratio of the projection length of the three-dimensional boundary projection on the image plane to the length of the spatially overlapping part in the image boundary segment with the same direction. The ratio is used as the current viewpoint boundary support. The higher the ratio, the greater the coverage of the three-dimensional boundary by the image edge and the stronger the current viewpoint boundary support. (2) Scheme based on the sampling point matching ratio: uniformly sample discrete points along the three-dimensional boundary projection path, and count the proportion of image edge points with the same direction in the normal neighborhood of each sampling point. The ratio of the number of matching sampling points to the total number of sampling points is used as the current viewpoint boundary support. (3) Scheme based on the image edge response intensity: integrate along the three-dimensional boundary projection path on the image edge response map or phase consistency map, and use the accumulated edge response intensity or average structure metric as the current viewpoint boundary support.

[0056] Step S142: Based on the homo-source visibility of the 3D boundary in the candidate boundary pair under the current spatiotemporal window, obtain the homo-source visibility range of the current window; wherein, the homo-source visibility range of the current window is used to quantify the proportion of the part in the 3D boundary covered by the obtained image edge that is consistent with the point cloud scanning state and is not occluded in the current field of view.

[0057] The aforementioned visible range of the current window is used to compare the structural comparability results of the current viewpoint boundary support level. It can be used to further determine whether the 3D boundary covered by the obtained image edge is consistent with the point cloud scanning state and is not occluded under the current shooting window and field of view. It is understandable that in the inspection of building facades, there is a time interval between the point cloud scanning time and the image shooting time. During this interval, there may be changes in components, temporary occlusion, or local construction, which will cause the 3D boundary in the point cloud to no longer have the same source visibility when the image is captured. At the same time, the structure in front of the camera's line of sight, such as balconies, eaves, or scaffolding, will block the rear boundary, making it impossible to see the 3D boundary recorded by the point cloud in the image. If the above-mentioned differences in spatiotemporal visibility conditions are not identified, the spatiotemporal mismatch residual caused by the inconsistency between the scanning time and the shooting time, as well as the invisible residual caused by the occlusion in front, will be mistakenly regarded as spatial registration deviation and included in the optimization, causing the overall registration result to deviate from the true spatial relationship. The purpose of obtaining the currently visible range of the same source within the current window in this embodiment of the invention is to: subtract the state change interval and the forward occlusion interval from the boundary supported by the obtained image edge, so that the remaining boundary length represents the part that is consistent with the point cloud scanning state and is actually visible in the current field of view, ensuring that only the boundary segments that are visible from the same source are included in the calculation of the subsequent degree of correction of the continuous residual of the same boundary, thus avoiding the erroneous interference of spatiotemporal mismatch residuals on registration optimization. An optional scheme for obtaining the currently visible range of the same source within the window is described below: Optionally, step S142 includes: for each current viewpoint imaging boundary segment, reading the scanning time period and image capture time, and determining the state change interval based on the component changes, temporary occlusions, and local construction records that overlap with the projection range of the corresponding boundary segment between the scanning time period and the capture time; reading the camera line of sight and the point cloud depth layer, checking whether there is a front structure segment in front of the corresponding boundary segment along the camera line of sight, and marking the overlapping part as a front occlusion interval when the projection range of the front structure segment overlaps with the corresponding boundary segment and the depth of the front structure segment is less than the depth of the corresponding boundary segment; determining the state change length of each state change interval and the front occlusion length of each front occlusion interval; determining the invalid overlap length between the state change interval and the front occlusion interval based on the state change length and the front occlusion length; determining the same-source visible length of the current window based on the total length of the imaging support and the invalid overlap length; and obtaining the same-source visible range of the current window based on the same-source visible length of the current window and the total length of the imaging support.

[0058] The state change interval reflects the visible state changes of the building facade between the scanning period and the image capture time. For example, during point cloud scanning, window opening boundaries may be unobstructed, while during image capture, the same area may be covered by suspended platforms or scaffolding; during point cloud scanning, the wall behind a balcony may be fully visible, while during image capture, some components may have been replaced. These component changes, temporary obstructions, and local construction records are mapped onto the corresponding imaging boundary segments based on timestamps and geometric locations, forming the state change interval. (State change length) The length of the union of each state change interval on the imaging boundary segment is obtained by calculating the total range of different sources of the three-dimensional boundary and the two-dimensional boundary due to time factors.

[0059] The foreground occlusion region reflects the occlusion effect of foreground structures on the rear boundary along the camera's line of sight. The point cloud depth layer is read along the camera's line of sight, checking for balconies, eaves, pipelines, scaffolding, or other foreground structure fragments in front of the imaging boundary segment. When the projected area of ​​the foreground structure fragment overlaps with the imaging boundary segment on the image plane, and the depth value of the foreground structure fragment in the point cloud depth layer is less than the depth value of the imaging boundary segment, it indicates that the foreground structure is located between the camera and the target boundary, and the overlapping area is marked as the foreground occlusion region. Foreground occlusion length. The length of the union of each forward occlusion interval on the imaging boundary segment is obtained by calculating the total range of the three-dimensional boundary that is not visible in the current image due to line-of-sight occlusion.

[0060] The state change interval and the forward occlusion interval may spatially overlap on the imaging boundary segment. This overlapping portion is only calculated once during subtraction and is recorded as the invalid overlap length. Overlapping invalid length This avoids redundant deductions for state changes and foreground occlusions within the same boundary region, ensuring that the calculation of the visible length of the same source is neither omitted nor repeated. Current window visible length of the same source. By measuring the total length of the imaging support Subtracting the invalid length of overlap The effective length of the actual homologous visible edge in the three-dimensional boundary covered by the edge of the obtained image is obtained.

[0061] Current window same-origin visible range Obtained using the following formula:

[0062] In the formula, For the first The current window's homologous visible range for each candidate boundary pair. For dimensionless parameters, The higher the value of , the greater the proportion of the visible part belonging to the current shooting window and the current field of view in the already imaged boundary; The invalid length of the overlap between the state change interval and the forward occlusion interval; This represents the total length of the imaging support. Units and The units are the same. The above-mentioned visible range of the current window is of the same origin. The calculation formula quantifies the proportion of the co-originating visible portion within the supported boundary by using the ratio of the overlapping invalid length to the total length of the imaging support. If the total length of the imaging support... If the value is zero, then the candidate boundary pair is not included in the calculation of the same-origin visible range of the current window. Therefore, There is no need to consider the case where the denominator is zero.

[0063] The determination of the aforementioned state change intervals is not limited to construction records and manual annotations. Component changes, temporary occlusions, and local construction records can be automatically extracted through multi-temporal image change detection. By comparing the pixel differences between the reference image during the scanning period and the image at the time of capture, the state change areas that occurred during this period are identified and located, and then the change areas are mapped onto the imaging boundary segment. The determination of the foreground structural segments can also be based on a 3D mesh model or a voxelized depth map. By using ray tracing along the camera's line of sight, structural surfaces that intersect with the imaging boundary segment and have a smaller depth are detected, improving the geometric accuracy of occlusion judgment.

[0064] The current window's common-source visible range, based on the current viewpoint imaging boundary segment, further deducts state changes between the scanning period and the shooting time, as well as forward occlusion in the camera's line-of-sight direction. This transforms the structurally comparable results of the current viewpoint boundary support into common-source visible results. The current window's common-source visible range distinguishes between uncorrectable residuals caused by spatiotemporal inconsistencies and correctable residuals caused by spatial deviations in the boundary length dimension. This provides a length benchmark for subsequently determining residual continuity only at the common-source visible boundary, preventing spatiotemporal mismatched residuals from entering the calculation of the correction degree of continuous residuals at the same boundary.

[0065] It should be noted that the embodiments of the present invention can also adopt one of the following schemes to obtain the same source visible range of the current window: (1) Scheme based on depth buffer visibility judgment: Project the point cloud onto the image plane to generate a depth map, compare the projection depth of the three-dimensional boundary point with the depth value of the corresponding pixel position in the depth map, if the projection depth of the boundary point is greater than the depth map value, it is determined that the point is occluded by the structure in front, the occluded part is deducted from the total length of the imaging support and the influence of the temporal state change record is eliminated to obtain the same source visible range of the current window. (2) Scheme based on ray tracing occlusion detection: Emit a ray from the optical center of the camera to the three-dimensional boundary point and detect whether the ray intersects with the triangular mesh of the structure in front. If there is an intersection and the depth of the intersection is less than the depth of the boundary point, it is determined that the boundary point is occluded, the length of the occluded part is subtracted from the length of the temporal state change interval by union, and the same source visible range of the current window is obtained. (3) Scheme based on semantic mask region subtraction: Semantic segmentation is performed on the image to obtain the two-dimensional mask of the front structure such as balcony, eaves and scaffolding. It is determined whether the three-dimensional boundary projection falls into the mask region of the front structure. The state change region determined by the change detection record between the scanning time period and the shooting time is performed by the union operation and then subtracted from the total length of the imaging support to obtain the same source visible range of the current window.

[0066] Step S143: Based on the continuous distribution of the projection residuals on the same source visible boundaries of the candidate boundary pairs, obtain the degree of correction of the continuous residuals on the same boundary; wherein, the degree of correction of the continuous residuals on the same boundary is used to quantify the degree to which the projection residuals maintain the same direction and continuous amplitude distribution along the same source visible boundaries.

[0067] The aforementioned continuous residual correction level along the same boundary is used to further identify the distribution characteristics of projection residuals on the visible boundary of the current window, in order to distinguish between correctable residuals originating from camera pose or local projection deviations and uncorrectable residuals originating from image edge object switching. It is understandable that repetitive curtain wall grids, dense pipelines, and reflective textures in building facades can cause image edge matching objects to switch at adjacent positions, resulting in a reversal of direction or a jump in amplitude of the projection residuals; while spatial deviations caused by camera pose or local projection relationships maintain a continuous, unidirectional distribution along the same boundary segment. If the above residual distribution characteristics are not judged, the residual jumps caused by object switching will be mistakenly treated as spatial correction residuals and included in registration optimization, leading the camera pose to adjust in the wrong direction. The purpose of obtaining the correction degree of continuous residuals at the same boundary in this embodiment of the invention is to: check the consistency of the residual projection sign and the continuity of the residual amplitude along the visible boundary of the same origin; identify residual segments with consistent direction and continuous amplitude distribution as correctable residuals; and identify residual relationships with sign reversal or abrupt amplitude changes as residual jumps. This ensures that only continuous residuals that meet the unified spatial offset characteristics enter the registration optimization, while jump residuals caused by object switching are isolated as mismatch criteria. An optional scheme for obtaining the correction degree of continuous residuals at the same boundary is described below: Optionally, step S143 includes: reading the projected residual vector for residual sampling points within the visible length of the current window, and projecting the projected residual vector onto the local correction direction; reading the projection sign and residual amplitude of each residual sampling point sequentially along the boundary arc length; connecting adjacent residual sampling points with consistent projection signs and residual amplitude changes not exceeding the jump boundary to obtain correctable residual segments, and accumulating the boundary arc length of each correctable residual segment to obtain the total length of correctable residual segments; counting the total number of adjacent residual relationships on the visible boundary of the current window, marking adjacent residual relationships with reversed projection signs or residual amplitude changes exceeding the jump boundary as residual jump relationships, and accumulating the residual jump count to obtain the residual jump percentage; and obtaining the degree of correction of continuous residuals on the same boundary based on the total length of correctable residual segments, the visible length of the current window, and the residual jump percentage.

[0068] The local correction direction is taken as the normal direction of the boundary segment in the image plane. This direction is perpendicular to the extension direction of the boundary segment in the image plane, representing the effective direction in which camera pose or local projection correction can push the point cloud projection boundary closer to the image boundary. The projection residual vector is projected onto this local correction direction, decomposing the two-dimensional offset vector into an effective component along the correction direction and an ineffective component along the boundary tangent. Only the effective component is retained to determine the spatial correction characteristics of the residual. This projection operation allows the residual analysis to focus on the directional dimension that actually contributes to registration optimization, eliminating the interference of the offset along the boundary tangent on the continuity judgment.

[0069] The projection sign and residual amplitude of each residual sampling point are read sequentially along the boundary arc length. The projection sign represents the orientation of the residual vector in the local correction direction, and the residual amplitude represents the magnitude of the deviation. When the projection signs of adjacent residual sampling points are consistent and the change in residual amplitude does not exceed the jump boundary, it indicates that the residuals at these adjacent locations originate from the same direction and have gradually changing amplitudes, thus connecting them into a correctable residual segment. The total length of the correctable residual segment is... The residual distribution range that conforms to the continuous unidirectional spatial migration characteristics on the visible boundary of the same origin is obtained by accumulating the boundary arc length of each correctable residual segment.

[0070] Count the total number of adjacent residual relationships on the visible boundary of the same source in the current window. This total represents the total number of relationships between adjacent residual sampling points on the same visible boundary. When the projection sign of adjacent residuals reverses or the residual amplitude changes beyond the jump boundary, it is recorded as a residual jump relationship, indicating a switch in the source of the residual at that adjacent location, possibly caused by repeated segmentation, reflective edges, or erroneous boundaries. Cumulative Residual Jump Count It can characterize the total number of jumps originating from the residual source. Percentage of residual jumps. Obtained using the following formula:

[0071] In the formula, For the first The percentage of residual jumps for each candidate boundary pair This is a dimensionless parameter; a higher value indicates that image edge object switching is more likely to occur on the currently visible boundary. For residual jump variables, It is a dimensionless parameter; This represents the total number of adjacent residual relationships. It is a dimensionless parameter.

[0072] Correction degree of continuous residuals at the same boundary Obtained using the following formula:

[0073] In the formula, For the first The degree of correction for continuous residuals of candidate boundary pairs within the same boundary. For dimensionless parameters, The higher the value, the more continuous the residuals on the currently visible boundary are, and the more consistent they are with the unified spatial deviation caused by the camera pose or local projection relationship. The total length of the correctable residual segment; The length of the current window that is visible to the same source; The percentage of residual jumps; This is the normalization function.

[0074] The aforementioned jump boundary can be determined based on the average residual amplitude within the segment. That is, the average residual amplitude of all residual sampling points on the visible boundary of the current window is taken as the benchmark, and adjacent relationships where the residual amplitude changes beyond this average are judged as jumps. The jump boundary can also be determined based on the standard deviation or median of the residual amplitude. By adaptively setting the jump threshold through statistical distribution characteristics, the jump judgment can be adapted to the overall level of residual amplitude on different boundary segments.

[0075] In addition to using the normal direction of the boundary segment in the image plane, the local correction direction can also use the projection direction of the translation or rotation components in the camera pose optimization parameters onto the image plane, so that the local correction direction corresponds to the actual registration optimization degree of freedom.

[0076] It should be noted that the embodiments of the present invention can also adopt one of the following schemes to obtain the degree of continuous residual correction at the same boundary: (1) Scheme based on residual direction clustering: cluster the residual vectors on the same visible boundary according to the direction, and count the proportion of the number of residual sampling points in the main direction cluster to the total number of residual sampling points. Use this proportion as the degree of continuous residual correction at the same boundary. (2) Scheme based on residual amplitude change rate statistics: calculate the residual amplitude change rate between adjacent residual sampling points, and count the proportion of the number of adjacent relationships with a change rate lower than the preset change rate threshold to the total number of adjacent relationships. Use this proportion as the degree of continuous residual correction to obtain the degree of continuous residual correction at the same boundary.

[0077] Step S144: Based on the current viewpoint boundary support level, the current window same-source visible range, and the same-boundary continuous residual correction level, obtain the spatial correction qualification status of the projection residual.

[0078] The following describes an optional implementation of step S144 above: Optionally, step S144 includes: based on the comparison result of the current viewpoint boundary support degree of each candidate boundary pair with the viewpoint boundary, determining the spatial correction eligibility state of candidate boundary pairs whose current viewpoint boundary support degree is lower than the viewpoint boundary as not having spatial correction eligibility state; for candidate boundary pairs whose current viewpoint boundary support degree reaches the viewpoint boundary, based on the comparison result of the current window's same-origin visible range with the spatiotemporal boundary, determining the spatial correction eligibility state of candidate boundary pairs whose current window's same-origin visible range is lower than the spatiotemporal boundary as a spatiotemporal mismatch state; for candidate boundary pairs whose current window's same-origin visible range reaches the spatiotemporal boundary, based on the comparison result of the same-boundary continuous residual correction degree with the correction boundary, determining the spatial correction eligibility state of candidate boundary pairs whose same-boundary continuous residual correction degree reaches the correction boundary as having spatial correction eligibility state, and determining the spatial correction eligibility state of candidate boundary pairs whose same-boundary continuous residual correction degree does not reach the correction boundary as not having spatial correction eligibility state.

[0079] The aforementioned viewpoint boundary is a threshold parameter used to determine whether the support level of the current viewpoint boundary reaches the structural comparability basis. This boundary divides candidate boundary pairs into two categories: those with stable image edge support and those without stable image edge support. When the support level of the current viewpoint boundary is lower than this boundary, it indicates that the 3D boundary does not have sufficient 2D edge coverage in the same direction under the current image viewpoint. The candidate boundary pair does not have a structural comparability basis as a spatial registration boundary, and its spatial correction eligibility status is directly determined as not having a spatial correction eligibility status, and it does not enter the subsequent calculation of the same source visible range and the judgment of the degree of continuous residual correction.

[0080] The aforementioned spatiotemporal boundary is a threshold parameter used to determine whether the visible range of the current window meets the standard for visible range. This boundary follows the screening results of the viewpoint boundary and further judges the spatiotemporal consistency of candidate boundary pairs whose support for the current viewpoint boundary meets the viewpoint boundary. When the visible range of the current window is lower than this boundary, it indicates that the proportion of the imaged boundary affected by state changes between the scanning period and the shooting time or by occlusion in front of the camera's line of sight is too large. The projection residuals on this candidate boundary pair mainly come from the inconsistency of spatiotemporal visibility conditions rather than spatial registration deviation. Its spatial correction eligibility state is determined to be a spatiotemporal mismatch state. The residuals in this state are isolated as spatiotemporal mismatch residuals and do not participate in global spatial optimization.

[0081] The aforementioned correction boundary is a threshold parameter used to determine whether the correction degree of continuous residuals along the same boundary reaches the correctable standard. This boundary performs a final screening on candidate boundary pairs whose visible ranges from the same source in the current window reach the spatiotemporal boundary, judging whether the residuals on the visible boundary from the same source exhibit a continuous and unidirectional distribution. When the correction degree of continuous residuals along the same boundary reaches this boundary, it indicates that the projection residuals maintain a consistent direction and continuous amplitude along the boundary, conforming to the unified spatial offset characteristics caused by camera pose or local projection deviation. The spatial correction eligibility status of this candidate boundary pair is determined to be eligible for spatial correction, and its correctable residual segments are included in the spatial registration optimization set. When the correction degree of continuous residuals along the same boundary does not reach this boundary, it indicates that the residuals have a reversed direction or a jump in amplitude, possibly caused by object switching. Its spatial correction eligibility status is determined to be ineligible for spatial correction.

[0082] Spatial correction eligibility states include three categories: lack of spatial correction eligibility, spatiotemporal mismatch, and eligibility. The lack of spatial correction eligibility corresponds to candidate boundary pairs where the current viewpoint boundary support is lower than the viewpoint boundary, and candidate boundary pairs where the current window's co-originating visible range reaches the spatiotemporal boundary but the continuous residual correction level of the same boundary does not reach the correction boundary. The projection residuals of these two types of boundary pairs are not eligible for spatial correction and are isolated from registration optimization. The spatiotemporal mismatch state applies to candidate boundary pairs where the current viewpoint boundary support reaches the viewpoint boundary but the current window's co-originating visible range is lower than the spatiotemporal boundary. The projection residuals of these boundary pairs originate from differences in spatiotemporal visibility conditions and are isolated, with occlusion verification, resampling, or time window adjustment flags output. The eligibility for spatial correction applies to candidate boundary pairs that simultaneously reach the viewpoint boundary, spatiotemporal boundary, and correction boundary. The projection residuals of these boundary pairs are eligible for spatial correction and are used to calculate camera pose correction or local projection correction.

[0083] Viewpoint boundaries, spatiotemporal boundaries, and correction boundaries can be determined by traversing candidate boundaries using qualified registration samples, manually reviewed samples, or calibration data. Qualified registration samples are derived from historical successful registration cases. By statistically analyzing the distribution characteristics of the current viewpoint boundary support, the current window's homogeneous visibility range, and the degree of correction of continuous residuals within the same boundary for candidate boundary pairs in successful registration cases, thresholds that can distinguish between correctable and uncorrectable boundaries are extracted. Manually reviewed samples are derived from the results of professional personnel's manual judgments on the correction eligibility of candidate boundary pairs. By quantifying expert experience into boundary thresholds, the blind spots of purely data-driven methods in complex scenarios are compensated for. Calibration data comes from precise registration experiments under controlled conditions. Candidate boundary pairs are established using cameras with known precise poses and point cloud data. The theoretical distribution of their various indicators is analyzed to determine baseline boundary values ​​applicable to ideal conditions.

[0084] Viewpoint boundary, spatiotemporal boundary, and correction boundary can form a progressive screening framework, with the screening result of the previous layer directly determining the processing path of the next layer. Candidate boundary pairs whose current viewpoint boundary support is lower than that of the viewpoint boundary are directly excluded from subsequent screening, preventing boundary residuals lacking image edge support from entering subsequent calculations. Candidate boundary pairs whose current viewpoint boundary support reaches that of the viewpoint boundary but whose current window's homologous visible range is lower than that of the spatiotemporal boundary enter the spatiotemporal mismatch processing path, and their residuals are isolated and a verification mark is output. Candidate boundary pairs that simultaneously reach both the viewpoint boundary and the spatiotemporal boundary enter the correction boundary judgment stage; only candidate boundary pairs whose continuous residual correction reaches the correction boundary are ultimately eligible for spatial correction.

[0085] Step S150: Filter projection residuals based on spatial correction eligibility status, and perform registration optimization based on correctable projection residuals to obtain spatiotemporal registration results.

[0086] Optionally, step S150 includes: adding the projection residuals of candidate boundary pairs with spatial correction eligibility status to the spatial registration optimization set; calculating the camera pose correction or local projection correction based on the projection residuals in the spatial registration optimization set, and updating the registration relationship between the image and the point cloud; isolating the projection residuals of candidate boundary pairs with spatial correction eligibility status of spatiotemporal mismatch or without spatial correction eligibility status as uncorrectable projection residuals, and not participating in global spatial optimization; outputting occlusion verification, resampling, or time window adjustment markers for structural segments with spatial correction eligibility status of spatiotemporal mismatch; and outputting the spatial correction eligibility status and registration status of each structural segment.

[0087] The aforementioned spatial registration optimization set consists of correctable projection residual segments of candidate boundary pairs that meet the spatial correction eligibility criteria. This set only includes residuals that have been confirmed to meet uniform spatial offset characteristics after a three-level progressive screening process based on the current viewpoint boundary support level, the visible range of the current window's origin, and the correction level of continuous residuals along the same boundary. Uncorrectable residuals caused by viewpoint compression, spatiotemporal mismatch, or object switching are excluded. This screening mechanism ensures that the residuals entering the optimization set are statistically spatially consistent, avoiding the contamination of the optimization objective function by abnormal residuals and improving the robustness of registration parameter estimation.

[0088] The projection residuals in the spatial registration optimization set are optimized to obtain either camera pose correction or local projection correction. Camera pose correction minimizes the overall offset between the point cloud projection boundary and the image boundary by adjusting the camera's spatial position and orientation in the architectural coordinate system, suitable for unified registration correction in global facade scenes. Local projection correction adjusts the local projection relationship of a specific facade segment, aligning the point cloud projection boundary within that segment with the image boundary locally, suitable for scenarios with non-rigid deformation or segmented scanning errors. Both corrections can be solved using optimization frameworks such as least squares, bundle adjustment, or iterative nearest point. The registration relationship between the image and the point cloud is updated by minimizing the weighted sum of squares of the residuals in the spatial registration optimization set. It should be noted that the aforementioned least squares, bundle adjustment, and iterative nearest point optimization frameworks are mature existing technologies; their implementation methods and working principles can be found in related technologies, and will not be elaborated further in this embodiment.

[0089] Projection residuals of candidate boundary pairs whose spatial correction eligibility is spatiotemporally mismatched or not eligible are isolated as uncorrectable projection residuals. These residuals do not participate in global spatial optimization and are not included in the objective function of camera pose or local projection, thus preventing them from erroneously influencing the registration results. For structural segments corresponding to spatiotemporally mismatched states, the system outputs occlusion verification, re-sampling, or time window adjustment markers, indicating that there is a state change or forward occlusion between the scanning period and the shooting time. Registration requires obtaining visible data from the same source through supplementary acquisition, adjusting the shooting window, or manual verification. Residuals not eligible for spatial correction indicate that the boundary lacks stable image support at the current viewpoint or that object switching exists, requiring verification using images from other viewpoints or higher resolution images.

[0090] The output after registration optimization includes the point cloud structure number, image boundary number, current viewpoint support status, current window same-origin visibility status, correctable residual segment, spatiotemporal mismatch residual segment, and final registration status for each structural segment. The point cloud structure number and image boundary number establish an index relationship between 3D structural units and 2D visual units, allowing image appearance information such as cracks, stains, and detachments to be traced back to specific structural segments in the point cloud. The current viewpoint support status and current window same-origin visibility status record the eligibility determination results of each segment during the screening process, providing a basis for subsequent quality assessment. The correctable residual segment and spatiotemporal mismatch residual segment respectively identify the residual distribution range participating in registration optimization and the isolated segment on each segment. The final registration status area is assigned quasi-stable segments and residual isolated segments.

[0091] like Figure 3 As shown, based on the same inventive concept, embodiments of the present invention also provide a spatiotemporal registration system 200 for building facade point clouds and images, comprising: The data acquisition module 210 is used to acquire point cloud data and image data of the building facade; The fragment extraction module 220 is used to extract point cloud structure fragments and image boundary fragments from point cloud data and image data, respectively. The boundary matching module 230 is used to match the three-dimensional boundary of the point cloud structure segment with the image boundary segment to establish candidate boundary pairs; The residual and qualification acquisition module 240 is used to acquire the projection residual between corresponding points in the candidate boundary pair and the spatial correction qualification status of the projection residual. The screening and registration module 250 is used to screen projection residuals based on spatial correction eligibility status and perform registration optimization based on correctable projection residuals to obtain spatiotemporal registration results.

[0092] It should be noted that the above-mentioned building facade point cloud and image spatiotemporal registration system 200 can realize any one of the functions of the building facade point cloud and image spatiotemporal registration method provided in the embodiments of the present invention. For the method embodiment section, please refer to the method embodiment section for the way each function is realized and the working principle. The embodiments of the present invention will not be repeated here.

[0093] This invention is now complete.

[0094] In summary, in this embodiment of the invention, point cloud data and image data of the building facade are acquired; point cloud structure fragments and image boundary fragments are extracted from the point cloud data and image data, respectively; the three-dimensional boundaries of the point cloud structure fragments are matched with the image boundary fragments to establish candidate boundary pairs; the projection residuals between corresponding points in the candidate boundary pairs and the spatial correction eligibility status of the projection residuals are acquired; the projection residuals are filtered based on the spatial correction eligibility status, and registration optimization is performed based on the correctable projection residuals to obtain the spatiotemporal registration result.

[0095] This invention determines the spatial correction qualification of projection residuals by sequentially judging the current viewpoint boundary support degree, the current window homogeneous visible range, and the continuous residual correction degree of the same boundary for candidate boundary pairs. This allows correctable residuals to enter the registration optimization while isolating spatiotemporally mismatched residuals, avoiding the erroneous pull of uncorrectable residuals on the overall registration. On the other hand, it reduces registration deviations caused by window openings, balcony protrusions, eaves, pipelines, scaffolding obstructions, and repeated curtain wall segmentation, ensuring that the registration results of point clouds and images conform to the true spatiotemporal relationship between the current image field of view and the point cloud scanning segment. Furthermore, it transforms projection residuals that do not qualify for spatial correction into occlusion verification, re-sampling, or time window adjustment markers, providing a directional basis for subsequent local data acquisition and temporal verification, avoiding information loss caused by directly discarding uncorrectable residuals. Finally, it improves the accuracy of mapping image textures, inspection marks, and appearance changes to point cloud structural segments, ensuring that information on defects such as cracks, stains, and peeling can be accurately mapped to the true location of the three-dimensional facade.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for spatiotemporal registration of point clouds and images of building facades, characterized in that, The method includes: Acquire point cloud data and image data of the building facade; Extract point cloud structure fragments and image boundary fragments from the point cloud data and the image data, respectively; The three-dimensional boundaries of the point cloud structure fragments are matched with the image boundary fragments to establish candidate boundary pairs; Obtain the projection residuals between corresponding points in the candidate boundary pair and the spatial correction eligibility status of the projection residuals; The method for obtaining the spatial correction eligibility status includes: Based on the image edge support of the 3D boundary in the candidate boundary alignment under the current viewpoint, the degree of boundary support under the current viewpoint is obtained. This degree of boundary support is used to quantify the extent to which the 3D boundary achieves 2D edge coverage in the same direction under the current image viewpoint, including: The ordered sampling pairs in the candidate boundary pairs are traversed in order along the boundary arc length. When there is an image edge sampling point in the normal neighborhood of the point cloud projection sampling point and the edge direction difference between the point cloud projection sampling point and the image edge sampling point does not exceed the direction boundary, the sampling point is marked as a support sampling point in the same direction. Connect consecutive adjacent support sampling points in the same direction along the boundary arc length to obtain the current viewpoint imaging boundary segment; The total length of the point cloud projection boundary is obtained by accumulating the arc lengths of the candidate boundary midpoint point cloud projection boundary point column; For each current viewpoint imaging boundary segment, the projected arc length between the support sampling points in the same direction within the segment is accumulated, the overlapping length of the image edges in the same direction is obtained, and the overlapping lengths of the image edges in the same direction of each segment are accumulated to obtain the total length of the imaging support. Based on the degree of deviation between the three-dimensional boundary unit extension direction of each current viewpoint imaging boundary segment and the camera's line of sight, the viewpoint compression deviation is obtained. Based on the aforementioned viewpoint compression deviation, the overlap length of the image edges in the same direction of the corresponding imaging boundary segment is adjusted to obtain the effective support length of the current viewpoint. Based on the effective support length of the current viewpoint and the total length of the point cloud projection boundary, the degree of boundary support of the current viewpoint is obtained; Based on the homo-source visibility of the 3D boundary in the candidate boundary alignment under the current spatiotemporal window, the homo-source visibility range of the current window is obtained; wherein, the homo-source visibility range of the current window is used to quantify the proportion of the part in the 3D boundary covered by the obtained image edge that is consistent with the point cloud scanning state and is not occluded in the current field of view. Based on the continuous distribution of the projection residuals on the same source visible boundaries of the candidate boundary pairs, the degree of correction for continuous residuals on the same boundary is obtained; wherein, the degree of correction for continuous residuals on the same boundary is used to quantify the degree to which the projection residuals maintain a consistent direction and a continuous amplitude distribution along the same source visible boundaries. Based on the current viewpoint boundary support level, the current window homogeneous visibility range, and the homogeneous boundary continuous residual correction level, the spatial correction eligibility status of the projection residual is obtained; the projection residual is filtered based on the spatial correction eligibility status, and registration optimization is performed based on the correctable projection residual to obtain the spatiotemporal registration result.

2. The method for spatiotemporal registration of point clouds and images of building facades according to claim 1, characterized in that, The step of obtaining the homologous visibility range of the current window based on the homologous visibility of the 3D boundaries in the candidate boundary alignment within the current spatiotemporal window includes: For each current viewpoint imaging boundary segment, the scanning time period and image capture time are read, and the state change range is determined based on the component changes, temporary occlusions and local construction records that overlap with the projection range of the corresponding boundary segment between the scanning time period and the capture time. Read the camera line of sight and the point cloud depth layer, check whether there is a front structure segment in front of the corresponding boundary segment along the camera line of sight. When the projection range of the front structure segment overlaps with the corresponding boundary segment and the depth of the front structure segment is less than the depth of the corresponding boundary segment, mark the overlapping part as the front occlusion area. The state change length of each of the state change intervals and the forward occlusion length of each of the forward occlusion intervals are determined respectively; Based on the state change length and the forward occlusion length, determine the invalid overlap length between the state change interval and the forward occlusion interval; Based on the total length of the imaging support and the invalid overlap length, the visible length of the current window from the same source is determined; Based on the current window's visible length and the total length of the imaging support, the current window's visible range is obtained.

3. The method for spatiotemporal registration of point clouds and images of building facades according to claim 2, characterized in that, The step of obtaining the degree of correction for continuous residuals at the same boundary based on the continuous distribution of the projection residuals on the same visible boundary of the candidate boundary pair includes: For the residual sampling points within the same visible length of the current window, read the projection residual vector and project the projection residual vector onto the local correction direction; Read the projection sign and residual amplitude of each residual sampling point sequentially along the boundary arc length; Connect adjacent residual sampling points with the same projection sign and whose residual amplitude changes do not exceed the jump boundary to obtain correctable residual segments, and accumulate the boundary arc length of each correctable residual segment to obtain the total length of the correctable residual segment. Count the total number of adjacent residual relationships on the same visible boundary of the current window, mark the adjacent residual relationships with reversed projection sign or residual amplitude changes exceeding the jump boundary as residual jump relationships, accumulate the residual jump number, and obtain the residual jump percentage. Based on the total length of the correctable residual segment, the length of the current window with the same source visible, and the percentage of residual jumps, the degree of correction for continuous residuals at the same boundary is obtained.

4. The method for spatiotemporal registration of point clouds and images of building facades according to claim 3, characterized in that, The step of obtaining the spatial correction eligibility status of the projection residual based on the current viewpoint boundary support level, the current window homogeneous visibility range, and the homogeneous boundary continuous residual correction level includes: Based on the comparison results between the current viewpoint boundary support level and the viewpoint boundary of each candidate boundary pair, the spatial correction eligibility status of candidate boundary pairs whose current viewpoint boundary support level is lower than that of the viewpoint boundary is determined to be a state of not having spatial correction eligibility. For candidate boundary pairs whose current viewpoint boundary support reaches the viewpoint boundary, based on the comparison results between the current window's common-origin visible range and the spatiotemporal boundary, the spatial correction qualification state of candidate boundary pairs whose current window's common-origin visible range is lower than the spatiotemporal boundary is determined as a spatiotemporal mismatch state. For candidate boundary pairs whose visible range of the same source window reaches the spatiotemporal boundary, based on the comparison results of the continuous residual correction degree of the same boundary and the correction boundary, the spatial correction qualification status of candidate boundary pairs whose continuous residual correction degree of the same boundary reaches the correction boundary is determined as having spatial correction qualification status, and the spatial correction qualification status of candidate boundary pairs whose continuous residual correction degree of the same boundary does not reach the correction boundary is determined as not having spatial correction qualification status.

5. The method for spatiotemporal registration of point clouds and images of building facades according to claim 4, characterized in that, The process of filtering the projection residuals based on the spatial correction eligibility status and performing registration optimization based on the correctable projection residuals to obtain spatiotemporal registration results includes: Add the projection residuals of candidate boundary pairs that are spatially correctable to the spatial registration optimization set; Based on the projection residuals in the spatial registration optimization set, the camera pose correction or local projection correction is calculated, and the registration relationship between the image and the point cloud is updated. The projection residuals of candidate boundary pairs whose spatial correction eligibility states are spatiotemporal mismatch states or whose spatial correction eligibility states are not eligible states are isolated as uncorrectable projection residuals and do not participate in global spatial optimization. For structural segments whose spatial correction eligibility status is spatiotemporal mismatch, output occlusion verification, re-sampling, or time window adjustment markers; Output the spatial correction qualification status and registration status of each structural segment.

6. The method for spatiotemporal registration of point clouds and images of building facades according to any one of claims 1-5, characterized in that, The step of obtaining the projection residuals between corresponding points in the candidate boundary pair includes: The cumulative arc lengths of the point cloud projection boundary point columns and the image boundary point columns in the candidate boundary pairs are calculated respectively, and their respective cumulative arc lengths are normalized. Interpolate points on the point cloud projection boundary point column and the image boundary point column according to the same normalized arc length position to form an ordered sampling pair; wherein, the ordered sampling pair includes point cloud projection boundary sampling points and image boundary sampling points at the same normalized arc length position; The projection residual is obtained based on the two-dimensional offset vector between the point cloud projection boundary sampling points and the image boundary sampling points.

7. The method for spatiotemporal registration of point clouds and images of building facades according to any one of claims 1-5, characterized in that, The step of extracting point cloud structure fragments and image boundary fragments from the point cloud data and the image data, respectively, includes: The point cloud data is divided into structural segments based on similar normals and similar distances from points to local planes. Three-dimensional boundary points are extracted at abrupt changes in depth, curvature, or outer contours of adjacent segments. The three-dimensional boundary points are arranged along the boundary extension direction to obtain point cloud structural segments. Two-dimensional edge points are extracted from the image data, edge segments that are continuous in direction and close in distance at their endpoints are merged, and the merged edge segments are arranged in order of arc length from endpoint to endpoint to obtain image boundary segments. The step of matching the 3D boundary of the point cloud structure segment with the image boundary segment to establish candidate boundary pairs includes: Project the three-dimensional boundary point array of the point cloud structure fragment onto the image plane; For each of the image boundary segments, select point cloud projection boundary segments with similar directions and similar endpoint order within the adjacent projection range to establish candidate boundary pairs.

8. A spatiotemporal registration system for point clouds and images of building facades, characterized in that, include: The data acquisition module is used to acquire point cloud data and image data of the building facade; The fragment extraction module is used to extract point cloud structure fragments and image boundary fragments from the point cloud data and the image data, respectively. The boundary matching module is used to match the three-dimensional boundary of the point cloud structure segment with the image boundary segment to establish candidate boundary pairs; The residual and qualification acquisition module is used to acquire the projection residual between corresponding points in the candidate boundary pair and the spatial correction qualification status of the projection residual. The method for obtaining the spatial correction eligibility status includes: Based on the image edge support of the 3D boundary in the candidate boundary alignment under the current viewpoint, the degree of boundary support under the current viewpoint is obtained. This degree of boundary support is used to quantify the extent to which the 3D boundary achieves 2D edge coverage in the same direction under the current image viewpoint, including: The ordered sampling pairs in the candidate boundary pairs are traversed in order along the boundary arc length. When there is an image edge sampling point in the normal neighborhood of the point cloud projection sampling point and the edge direction difference between the point cloud projection sampling point and the image edge sampling point does not exceed the direction boundary, the sampling point is marked as a support sampling point in the same direction. Connect consecutive adjacent support sampling points in the same direction along the boundary arc length to obtain the current viewpoint imaging boundary segment; The total length of the point cloud projection boundary is obtained by accumulating the arc lengths of the candidate boundary midpoint point cloud projection boundary point column; For each current viewpoint imaging boundary segment, the projected arc length between the support sampling points in the same direction within the segment is accumulated, the overlapping length of the image edges in the same direction is obtained, and the overlapping lengths of the image edges in the same direction of each segment are accumulated to obtain the total length of the imaging support. Based on the degree of deviation between the three-dimensional boundary unit extension direction of each current viewpoint imaging boundary segment and the camera's line of sight, the viewpoint compression deviation is obtained. Based on the aforementioned viewpoint compression deviation, the overlap length of the image edges in the same direction of the corresponding imaging boundary segment is adjusted to obtain the effective support length of the current viewpoint. Based on the effective support length of the current viewpoint and the total length of the point cloud projection boundary, the degree of boundary support of the current viewpoint is obtained; Based on the homo-source visibility of the 3D boundary in the candidate boundary alignment under the current spatiotemporal window, the homo-source visibility range of the current window is obtained; wherein, the homo-source visibility range of the current window is used to quantify the proportion of the part in the 3D boundary covered by the obtained image edge that is consistent with the point cloud scanning state and is not occluded in the current field of view. Based on the continuous distribution of the projection residuals on the same source visible boundaries of the candidate boundary pairs, the degree of correction for continuous residuals on the same boundary is obtained; wherein, the degree of correction for continuous residuals on the same boundary is used to quantify the degree to which the projection residuals maintain a consistent direction and a continuous amplitude distribution along the same source visible boundaries. Based on the current viewpoint boundary support level, the current window homogeneous visible range, and the homogeneous boundary continuous residual correction level, the spatial correction eligibility status of the projection residual is obtained; The screening and registration module is used to screen the projection residuals based on the spatial correction eligibility status, and to perform registration optimization based on the correctable projection residuals to obtain the spatiotemporal registration result.

Images