Multi-source spatio-temporal registration positioning method and system for highway slope inspection

Abstract

The present application relates to the technical field of highway slope inspection, and the prior art cannot meet the actual needs of highway digital inspection and intelligent operation and maintenance, and provides a multi-source space-time registration positioning method and system for highway slope inspection, which includes multi-source joint correction of the current time prediction state of the inspection terminal; a near-field observation sector of the current time is constructed; a plurality of candidate spatial units are divided within the coverage range of the near-field observation sector, an abnormal area is identified and its boundary range and spatial center position are extracted, and are matched with the route center line of the current inspection section to obtain corresponding milepost information, the abnormal area is mapped to a standardized structured positioning result, and the positioning result of the abnormal area is evaluated in terms of the current point cloud registration quality, point cloud overlap, satellite positioning quality and terminal attitude stability, which can make the result directly serve the slope inspection filing, maintenance disposal distribution and review verification.

Description

Technical Field

[0001] This invention relates to the field of highway slope inspection technology, and in particular to a multi-source spatiotemporal registration and positioning method and system for highway slope inspection. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In highway infrastructure, slopes are widely distributed in road cuts, embankments, interchange connecting sections, and bridge-tunnel transition zones. They are subject to long-term effects from factors such as rainfall, snowmelt, weathering, engineering disturbances, and traffic loads, and are prone to problems such as water damage, gully erosion, collapse, landslides, abnormal protective structures, and drainage failures. In existing research and engineering practice, slope inspection is usually carried out along three routes: The first is traditional or semi-automatic inspection methods such as manual inspection, vehicle-mounted inspection, and drone inspection. The advantage of these methods is their mature deployment, but they generally suffer from insufficient coverage of steep slopes, spaces under bridges, areas obscured by vegetation, and close-range work surfaces. The second is slope anomaly detection methods based on image recognition or video recognition. These technologies have certain efficiency advantages in anomaly detection, but the inspection results often remain at the image level, making it difficult to directly form a spatial expression associated with mileage markers, slope zoning, elevation zones, and three-dimensional coordinates. The third is spatial mapping or positioning methods based on lidar, satellite positioning, and inertial measurement units. These technologies have strong spatial expression capabilities, but existing solutions mostly focus on single-device positioning, local mapping, or static measurement, and lack an integrated spatiotemporal registration and positioning mechanism of "inspection terminal-lidar-satellite positioning-scene prior" for highway slope inspection scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-source spatiotemporal registration and positioning method and system for highway slope inspection. The results can directly serve slope inspection documentation, maintenance and treatment assignment, and verification, thus making it more suitable for the digital, engineering-friendly, and pre-approval-friendly technical path of highway infrastructure.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a multi-source spatiotemporal registration and positioning method for highway slope inspection.

[0006] In one or more embodiments, a multi-source spatiotemporal registration and positioning method for highway slope inspection is provided, including: Simultaneously receive multi-source sensing data, including images from the inspection terminal's perspective, 3D point clouds from LiDAR, inertial measurement information, satellite positioning information, and prior data of the slope scene; Based on inertial measurement information, the continuous motion state of the inspection terminal in the slope scene is calculated. Under the absolute positioning constraints, lidar geometric registration constraints and slope linear prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, multi-source joint correction is performed to obtain the pose of the inspection terminal in the slope scene coordinate system. The three-dimensional point cloud of the lidar is uniformly mapped to the slope scene coordinate system, and then the near-field observation sector at the current moment is constructed based on the current pose, field of view and effective observation range of the inspection terminal. Within the coverage area of ​​the near-field observation sector, several candidate spatial units are divided. Based on the local elevation difference changes, point cloud density changes, and continuous observation attention values ​​of the inspection terminal for each candidate spatial unit, abnormal areas are identified and their boundary range and spatial center location are extracted. The spatial center of the abnormal area is matched with the centerline of the current inspection section to obtain the corresponding mileage marker information. Then, combined with the slope zoning rules and elevation zone division rules, the abnormal area is mapped into a standardized structured positioning result and its credibility is evaluated.

[0007] As one implementation method, the process of calculating the continuous motion state of the inspection terminal in a slope scenario based on inertial measurement information is as follows: Define the state vector of the inspection terminal at each epoch. Based on the posterior state of the previous epoch and the current inertial measurement information, use the inertial propagation function to make prior predictions on the state of the inspection terminal. During inertial propagation, inertial measurement information is recursively updated, and smoothing constraints are applied to attitude changes in adjacent epochs to suppress the impact of short-term attitude changes caused by slope uphill and downhill operations, vegetation obstruction and detours, and local sudden stop observations on attitude prediction, thereby obtaining the predicted state of the inspection terminal.

[0008] As one implementation method, the objective function corresponding to the pose of the inspection terminal in the slope scene coordinate system is the optimal pose estimate corresponding to the weighted sum of the absolute positioning constraint, the lidar geometric registration constraint, and the slope alignment prior constraint; where the absolute positioning constraint is the position residual of the inspection terminal; the lidar geometric registration constraint is the lidar geometric registration residual; and the slope alignment prior constraint is the slope prior consistency residual.

[0009] As one implementation method, candidate spatial cells are generated according to local meshes, voxel cells, or slope slices on the slope surface.

[0010] As one implementation method, the continuous observation attention value of the inspection terminal is generated by combining the dwell time, the number of times the line of sight is repeatedly pointed, or the edge trigger record.

[0011] As one implementation method, based on standardized structured positioning results, combined with lidar geometric registration residuals, point cloud overlap rate, satellite positioning solution quality factor, and inspection terminal attitude jitter, the reliability of positioning results in abnormal areas is assessed.

[0012] As one implementation method, when the credibility requirement is met, the location result of the abnormal area is taken as a valid inspection result and written into the background inspection log; when the credibility is insufficient, the location result of the abnormal area is marked as pending review, triggering a local backtracking correction process to readjust the pose solution and spatial mapping results of the inspection terminal.

[0013] A second aspect of the present invention provides a multi-source spatiotemporal registration and positioning system for highway slope inspection.

[0014] In one or more embodiments, a multi-source spatiotemporal registration and positioning system for highway slope inspection includes: The multi-source synchronous sensing module is used to synchronously receive multi-source sensing data, which includes images from the inspection terminal's perspective, three-dimensional point clouds from lidar, inertial measurement information, satellite positioning information, and prior data of the slope scene. The predictive state correction module is used to calculate the continuous motion state of the inspection terminal in the slope scene based on inertial measurement information. It performs multi-source joint correction under the absolute positioning constraints, lidar geometric registration constraints and slope linear prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, to obtain the pose of the inspection terminal in the slope scene coordinate system. The near-field observation sector construction module is used to uniformly map the three-dimensional point cloud of the lidar to the slope scene coordinate system, and then construct the near-field observation sector at the current moment based on the current pose, field of view and effective observation range of the inspection terminal. The abnormal region identification module is used to divide the near-field observation sector coverage area into several candidate spatial units. Based on the local elevation difference change, point cloud density change and continuous observation attention value of each candidate spatial unit, the abnormal region is identified and its boundary range and spatial center position are extracted. The anomaly area credibility assessment module is used to match the spatial center location of the anomaly area with the route centerline of the current inspection section to obtain the corresponding mileage station information. Then, combined with the slope zoning rules and elevation zone division rules, the anomaly area is mapped into a standardized structured positioning result and its credibility is assessed.

[0015] A third aspect of the present invention provides a computer-readable storage medium.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multi-source spatiotemporal registration and positioning method for highway slope inspection as described above.

[0017] A fourth aspect of the present invention provides an electronic device.

[0018] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the multi-source spatiotemporal registration and positioning method for highway slope inspection described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a multi-source spatiotemporal registration and positioning method for highway slope inspection. Based on inertial measurement information, the continuous motion state of the inspection terminal in the slope scene is calculated to obtain the predicted state of the inspection terminal at the current moment. Multi-source joint correction is then performed. According to the current pose, field of view, and effective observation range of the inspection terminal, a near-field observation sector for the current moment is constructed, and several candidate spatial units are divided within its coverage area. Abnormal areas are identified, and their boundary range and spatial center position are extracted. The spatial center position of the abnormal area is matched with the route centerline of the current inspection section to obtain the corresponding mileage station information. Then, combined with slope zoning rules and elevation zone division rules, the abnormal area is mapped into a standardized structured positioning result, and its credibility is evaluated. The result directly serves slope inspection filing, maintenance and disposal assignment, and review and verification, thus being more suitable for the digital, engineering, and pre-approval-friendly technical path of highway infrastructure. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of a multi-source spatiotemporal registration and positioning method for highway slope inspection according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of a multi-source spatiotemporal registration and positioning system for highway slope inspection according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the multi-coordinate system relationship and spatial mapping according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the spatial mapping and mileage association of abnormal candidate regions in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Terminology Explanation: Multi-source sensing data refers to a data set generated by two or more heterogeneous sensors or information sources and ultimately used for unified registration and positioning in the same slope inspection task. It includes at least one or more combinations of inspection terminal image sequences, lidar point clouds, inertial measurement unit (IMU) observations, global navigation satellite system (GNSS) observations, and prior knowledge of slope alignment.

[0026] Spatiotemporal registration refers to the process of unifying the time and space of data from different sources, frequencies, and coordinate systems. Time alignment ensures that different sensor data correspond to the same inspection time or the same time window, while spatial alignment ensures that different sensor data can be mapped to the same slope scene coordinate system for expression and calculation.

[0027] Pose: refers to the combination of the position and orientation of the inspection terminal in the slope scene. Position describes the terminal's three-dimensional coordinates in the scene, while orientation describes the terminal's orientation and rotation. Pose is a key intermediate variable for anchoring point cloud and image observations to the slope scene.

[0028] Slope Scene Coordinate System: This refers to a unified spatial coordinate system established with reference to the highway slope inspection object. It is used to carry the inspection point cloud, terminal trajectory, abnormal area boundaries, and structured archiving results. This coordinate system can be established from existing surveying results, point cloud maps, or the anchoring results of the first frame of the inspection.

[0029] Anomaly area: refers to a spatial object that requires further location and documentation during slope inspection, pointed to by information such as near-field observation clues, geometric abrupt changes, surface damage characteristics, abnormal slope undulations, abnormal drainage patterns, or abnormal protective structures. In this invention, this object is first represented as a candidate area, and then mapped into a structured result with boundaries, center point, mileage marker, and credibility.

[0030] Mileage correlation: This refers to the process of projecting or matching the three-dimensional position of anomaly areas in the slope scene coordinate system to the highway alignment reference system, thereby obtaining their corresponding station intervals, driving directions, slope zones, or facility locations. This process enables direct integration of inspection results with existing maintenance management records.

[0031] Reliability: refers to the quantitative measure of the reliability of the system's spatial positioning results for the current abnormal area. Reliability is not the confidence level of a single sensor, but a comprehensive index determined by factors such as registration residual, point cloud overlap rate, GNSS signal quality, terminal attitude stability, and prior consistency.

[0032] Slope inspection must not only meet the requirements of highway maintenance safety operation procedures for safe organization of maintenance operations, but also serve the requirements of highway technical condition assessment standards for technical condition assessment, automation of inspection, and informatization of assessment. Furthermore, it must comply with the overall requirements of highway engineering technical standards regarding the operational safety and facility compatibility of expressways. Therefore, slope inspection cannot merely remain at the level of "discovering anomalies"; it must also generate spatialized results that are verifiable, locatable, documentable, and capable of closed-loop circulation.

[0033] Based on existing engineering applications, the inconsistent time bases, sampling frequencies, and coordinate systems of multi-source data make it difficult to stably integrate inspection images, point clouds, and positioning information, resulting in insufficient ability to reproduce the location of abnormal areas. Existing solutions often separate anomaly detection from spatial positioning. While the front end can detect suspected problems, the back end struggles to directly obtain reliable 3D location, mileage markers, and slope zoning information, causing a break in the data chain between inspection and maintenance. In the typical complex scenario of highway slopes, factors such as obstruction, slope undulation, satellite positioning signal fluctuations, drastic changes in terminal attitude, and operational safety constraints make it difficult for a single positioning or identification solution to simultaneously achieve real-time performance, accuracy, robustness, and engineering feasibility, and thus cannot meet the actual needs of digital inspection and intelligent operation and maintenance of highways.

[0034] To address the problems in the background art, embodiments of the present invention provide a multi-source spatiotemporal registration and positioning method and system for highway slope inspection. According to Figure 1 The multi-source spatiotemporal registration and positioning method for highway slope inspection in this embodiment may include the following: Step 1: Simultaneously receive multi-source sensing data, which includes images from the inspection terminal's perspective, 3D point clouds from LiDAR, inertial measurement information, satellite positioning information, and prior data of the slope scene.

[0035] Before the commencement of highway slope inspection, a multi-source inspection and perception system is constructed, comprising an inspection terminal, lidar, inertial measurement unit, satellite positioning module, and a priori database of slope scenarios. The inspection terminal, lidar, inertial measurement unit, and satellite positioning module are each connected to an edge computing unit, which communicates with a wireless communication module and a backend management platform. The steps in the multi-source spatiotemporal registration and positioning method for highway slope inspection in this embodiment can be executed within the backend management platform. The backend management platform also pre-stores historical scenario models (i.e., priori data of slope scenarios), route and station information (i.e., slope zoning ledgers), such as... Figure 2 As shown.

[0036] In this embodiment, the inspection terminal is used to acquire the inspection terminal's perspective image sequence and attitude information during the near-field operation of the inspection personnel on the slope. The lidar is used to acquire three-dimensional point cloud information of the slope surface and its associated structures. The inertial measurement unit is used to output the angular velocity and linear acceleration observations during the inspection terminal's movement. The satellite positioning module uses the Global Navigation Satellite System (GNSS) to provide global position anchoring information. The slope scene prior database provides the route centerline, station information, slope zoning information, and historical scene model information for the corresponding inspection section. Through the above multi-source collaborative acquisition, a raw data foundation is provided for subsequent unified time reference construction, unified scene coordinate system registration, and spatial positioning of abnormal areas. During the inspection of the calendar Location, synchronous collection , , , as well as ;in, For inspection epoch index; For the first Each inspection team has a history of 100,000 yuan; For the calendar The following is a first-view image frame captured by the inspection terminal; For lidar coordinate system The following point cloud collection ； For the first The three-dimensional coordinates of a laser point in the lidar coordinate system; This represents the number of points in the point cloud for that epoch. For IMU in the epoch Output angular velocity and linear acceleration observations; For GNSS in epoch Output position, velocity, and solution quality observations; This step generates a set of prior information regarding the slope alignment, stationing zones, slope zoning, and scene model for the current inspection section. The output of this step is the original multi-source observation set, which provides input for unified time synchronization processing.

[0037] To address the timing drift issue caused by different sampling frequencies and inconsistent internal clocks of multi-source devices, a unified time reference is constructed based on hardware timing signals, device timestamps, or software synchronization signals. All original observations are then resampled to an inspection epoch sequence under this unified time reference. Above, a synchronous observation group is formed: ; in, For the calendar The synchronous observation group below; This provides a unified time reference for the inspection epoch sequence. For data streams with different sampling frequencies, time interpolation, nearest neighbor matching, or sliding time window alignment are used to generate synchronous observation results under a unified epoch, ensuring that subsequent inspection terminal pose estimation, point cloud projection, anomaly region extraction, and mileage correlation are all based on the same time reference.

[0038] Step 2: Based on inertial measurement information, the continuous motion state of the inspection terminal in the slope scene is calculated. Under the absolute positioning constraints, lidar geometric registration constraints and slope line shape prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, multi-source joint correction is performed to obtain the pose of the inspection terminal in the slope scene coordinate system.

[0039] The process of calculating the continuous motion state of the inspection terminal in a slope scenario based on inertial measurement information is as follows: Define the state vector of the inspection terminal at each epoch. Based on the posterior state of the previous epoch and the current inertial measurement information, use the inertial propagation function to make prior predictions on the state of the inspection terminal. Specifically, in obtaining the synchronous observation group Then, define the epoch. The state vector of the inspection terminal is: ; in, For the calendar The state vector of the inspection terminal below; For the calendar The inspection terminal below is in the slope scene coordinate system The position vector in; For the calendar The inspection terminal below is in the slope scene coordinate system The velocity vector in the middle; For the calendar The quaternion of the current attitude of the inspection terminal; For the calendar The accelerometer shows zero bias. For the calendar The gyroscope is at zero bias. This is the transpose of the matrix. Through the above state definition, the spatial position, speed, attitude changes, and inertial device errors of the inspection terminal during highway slope inspection are uniformly incorporated into the same state expression framework.

[0040] During inertial propagation, inertial measurement information is recursively updated, and smoothing constraints are applied to attitude changes in adjacent epochs to suppress the impact of short-term attitude changes caused by slope uphill and downhill operations, vegetation obstruction and detours, and local sudden stop observations on attitude prediction, thereby obtaining the predicted state of the inspection terminal.

[0041] Based on the posterior state of the previous epoch and current IMU observations By performing prior prediction on the status of the inspection terminal, we obtain: ; in, For the calendar The prior predicted state; This is the state propagation function constructed based on observations from the inertial measurement unit. This propagation process reflects the continuous motion trend of the inspection personnel as they walk, stop, turn, or observe the slope, thus providing the initial pose for the next step of multi-source joint correction.

[0042] In one implementation, the inertial propagation function can be expanded into a position propagation sub-model, a velocity propagation sub-model, an attitude propagation sub-model, and a zero-bias propagation sub-model. Let the epoch... The time interval between two adjacent inspection epochs, the IMU at epoch The output angular velocity observation is Linear acceleration observation is The prior propagation of the inspection terminal status can be expressed as: ; in, For the calendar The prior prediction value of the location of the inspection terminal; For the calendar The location of the inspection terminal below; For the calendar The speed of the inspection terminal is reduced; For the calendar The quaternion of the inspection terminal below; For quaternions The corresponding rotation matrix; For the calendar The prior prediction value of the speed of the inspection terminal under the given conditions; For the calendar The prior prediction value of the attitude quaternion of the inspection terminal; For the calendar The prior prediction of the accelerometer zero bias; For the calendar The prior prediction value of the gyroscope with zero bias; For the calendar The gyroscope is at zero bias. For the calendar The accelerometer shows zero bias. For the calendar The accelerometer shows zero bias. This is the vector of gravitational acceleration; Represents quaternion multiplication; This represents the exponential mapping of attitude increment quaternions obtained from the mapping of angular velocity increments; and Representing epochs The random walk terms of the accelerometer zero bias and gyroscope zero bias.

[0043] In one implementation, to suppress short-term attitude changes caused by slope uphill / downhill operations, vegetation obstruction, and sudden stops for observation, a smoothing constraint can be applied to attitude changes in adjacent epochs, which can be expressed as follows: ; in, For attitude change constraints between adjacent epochs; Represents the quaternion logarithmic mapping; It is a Euclidean norm. The attitude change constraint term can be used as a regularization term in subsequent joint optimization, or to suppress anomalous attitude jumps.

[0044] To improve the stability of motion propagation in highway slope inspection scenarios, the accelerometer is zero-biased during inertial propagation. and gyroscope zero bias Recursive updates are performed, and smoothing constraints are applied to attitude changes in adjacent epochs to suppress the impact of short-term attitude abrupt changes caused by slope operations, vegetation obstruction, and sudden stops for observation on pose prediction. After this step, the predicted state is output. This will be used for the next step of jointly introducing GNSS, lidar, and slope priors for correction.

[0045] The objective function corresponding to the pose of the inspection terminal in the slope scene coordinate system is the optimal pose estimate corresponding to the weighted sum of the absolute positioning constraint, the lidar geometric registration constraint, and the slope alignment prior constraint. Among them, the absolute positioning constraint is the position residual of the inspection terminal; the lidar geometric registration constraint is the lidar geometric registration residual; and the slope alignment prior constraint is the slope prior consistency residual.

[0046] The process of multi-source joint correction and unified scene pose solution is as follows: Obtaining the predicted state Subsequently, GNSS absolute positioning constraints, lidar geometric registration constraints, and slope alignment prior constraints were jointly introduced to perform multi-source joint correction on the predicted state, obtaining the coordinate system of the inspection terminal in the slope scene. Unified Posture .in, The coordinate system of the inspection terminal body; To the inspection terminal body coordinate system To the slope scene coordinate system The homogeneous transformation matrix. This pose matrix describes both the position of the inspection terminal in the slope scene and its orientation and attitude, and is the core intermediate result for subsequent point cloud mapping and observation space anchoring; The unified pose is obtained through the following objective function: ; in, For the calendar Optimal pose estimation; These are the GNSS constraint weighting coefficients; These are the constraint weighting coefficients for the lidar. The weighting coefficients for prior constraints on slopes; This refers to the GNSS position residual term; For the geometric registration residuals of the lidar; For the a priori consistency residuals of slope alignment and slope boundary; This represents a function for finding the optimal pose.

[0047] The weighting coefficients here can be determined based on the reliability of each observation source, its own noise level, scene adaptability, or historical calibration statistics, and can be dynamically adjusted during the inspection process based on GNSS solution quality, point cloud matching quality, and prior consistency. When the GNSS signal is good, increase the GNSS weight; when there is severe obstruction, decrease the GNSS weight; when the point cloud overlap is high and the geometry is stable, increase the lidar constraint weight; when the prior is reliable, increase the slope alignment prior weight.

[0048] By minimizing the above objective function, global positioning information, local geometric information, and route prior information are unified into the same optimization process, thus avoiding positioning drift of a single sensor in complex slope environments. To ensure that each residual term has a clear physical meaning in the highway slope scenario, the GNSS position residual term is further defined. for: ; in, The estimated position of the inspection terminal in the slope scene coordinate system; The location of the inspection terminal after coordinate transformation of GNSS observations; It is the Euclidean norm. This residual term is used to ensure that the pose of the inspection terminal does not deviate from the route segment position provided by satellite positioning in the global scope; Define the geometric registration residual term of lidar for: ; in, For the calendar The number of effective laser points participating in the registration calculation; For the calendar The next The three-dimensional coordinates of each effective laser point mapped to the slope scene coordinate system; This is the nearest neighbor projection operator from a point to the slope scene model or the reference point cloud of an adjacent frame. This residual term is used to measure the consistency between the current point cloud and the slope scene geometry, ensuring that the pose of the inspection terminal meets the local spatial mapping accuracy requirements.

[0049] It should be noted that in some other embodiments, the root mean square error can also be used for the lidar geometric registration residual term.

[0050] Define the a priori consistency residuals of slope alignment and slope boundary. for: ; in, To measure the lateral deviation of the inspection terminal position from the center curve of the highway alignment; This represents the inconsistency between the observation direction of the inspection terminal and the direction of the slope boundary. This residual term is used to constrain the trajectory of the inspection terminal to maintain consistency with the route direction, slope orientation, and slope zoning, preventing the pose calculation of the inspection terminal from deviating from the actual slope inspection area. After joint optimization, a unified pose is output. This is for subsequent steps to perform multi-source spatial projection.

[0051] Step 3: Based on the calibration relationship between the inspection terminal and the lidar, the lidar 3D point cloud is uniformly mapped to the slope scene coordinate system. Then, based on the current pose, field of view, and effective observation range of the inspection terminal, the near-field observation sector at the current moment is constructed.

[0052] To achieve a uniform posture Then, the fixed extrinsic parameter matrix between the inspection terminal and the lidar is pre-called. , will the calendar The laser point cloud below is from the lidar coordinate system Mapped to the slope scene coordinate system For any point The mapping result is expressed as: ; in, These are the coordinates of the scene points after being mapped to the slope scene coordinate system; To the lidar coordinate system To the inspection terminal body coordinate system The homogeneous transformation matrix. By performing the same extrinsic pose transformation on all points, a uniform scene point cloud can be obtained. This places the geometry of the slope surface, its auxiliary protective structures, and local slope undulations within a unified spatial reference.

[0053] After mapping the point cloud to the slope scene coordinate system, according to the unified pose By inspecting the terminal's internal parameters and field of view parameters, the observation sector for the current epoch is constructed. The observation sector describes the near-field slope area that is actually observable by the inspection personnel in their current posture, and its construction form is represented as follows: ; in, Operators are generated for the observed sectors; For the inspection terminal's field of view; The minimum effective observation distance; To determine the effective maximum observation distance, the observation sector is used. The construction of this method can anchor the observation range of the first-view image to the three-dimensional space of the slope without having to take the image recognition result itself as the main invention point. like Figure 3 As shown, point clouds in a unified scene and observation sector This process involves jointly constructing observable spatial objects on the slope in the current epoch, preserving both the spatial geometric accuracy of the lidar and the spatial directivity of the inspection terminal's near-field observations. This step outputs a unified scene point cloud. and near-field observation sector This provides spatial extent and geometric input for the next step of constructing anomaly candidate regions.

[0054] Step 4: Divide the near-field observation sector coverage area into several candidate spatial units. Based on the local elevation difference changes, point cloud density changes, and continuous observation attention values ​​of the inspection terminal for each candidate spatial unit, identify the abnormal areas and extract their boundary range and spatial center location. In the observation sector Construct candidate spatial units within the coverage area This allows for spatial partitioning of local slope anomalies. Candidate spatial cells can be generated based on local slope surface meshes, voxel cells, or slope slices, and their indices... Indicates the first Multiple candidate spatial units. By decomposing the observable space of a slope into multiple candidate units, a continuous point cloud scene can be transformed into an analytical object suitable for calculating the degree of local anomalies.

[0055] For each candidate spatial unit Calculate the local elevation change amount Local point density gradient and continuous observation attention value .in, Used to characterize the elevation fluctuation of the spatial unit relative to the neighboring units, so as to reflect the sudden changes in elevation caused by slope erosion, collapse, local rockfall or deformation of protective structures; It is used to characterize the degree of variation of the point cloud density of the unit in local space, so as to reflect the existence of abrupt edges, holes, shading zones or local density anomalies on the slope surface; The continuous observation attention value characterizes the intensity of continuous observation of the unit by the inspectors. It can be generated by combining the dwell time, the number of times the line of sight is repeatedly pointed, or the trigger record at the edge. Through these three quantities, both slope geometric anomaly clues and near-field inspection behavior clues are simultaneously integrated. Based on the above three quantities, construct candidate unit anomaly scores: ; in, For the first Anomaly scores for each candidate spatial unit; This is the weighting coefficient for the elevation change abruptness; These are the point density gradient weight coefficients; This represents the weighting coefficient for attention values ​​observed continuously. This anomaly score is used to comprehensively determine whether there are anomalous candidate features in a local spatial area of ​​the slope, rather than directly relying on label recognition from a single frame image.

[0056] The weighting coefficients in the anomaly scoring can be determined based on historical inspection sample statistics, training results of labeled anomaly samples, expert experience, or validation set optimization results, and can be adjusted according to slope type, vegetation occlusion degree, and inspection task requirements. In one embodiment, the above-mentioned anomaly scoring weights can be determined through historical sample statistical analysis; in another embodiment, they can also be determined through supervised learning, grid search, or cross-validation of known anomaly samples; in other embodiments, they can be preset by human experience and continuously corrected in the background operation.

[0057] When the candidate unit abnormal score satisfies When a candidate spatial unit is identified as an abnormal candidate region, its spatial boundary is output. and candidate center points .in, This is the threshold for abnormal scoring (those skilled in the art can set the specific value according to the actual situation); This represents the boundary of the anomaly candidate region in the slope scene coordinate system; This is the spatial center point of the anomaly candidate region. This step transforms the unified scene point cloud and observation space into anomaly candidate region objects, outputting a set of anomaly candidate regions. ; in, For the calendar The set of anomalous candidate regions is used as direct input for the next step of mileage association and structured representation.

[0058] Step 5: Match the spatial center location of the abnormal area with the center line of the current inspection section, such as... Figure 4 As shown, the corresponding mileage station information is obtained. Then, combined with the slope zoning rules and elevation zone division rules, the abnormal area is mapped into a standardized structured positioning result and its credibility is evaluated.

[0059] Based on standardized structured positioning results, and combined with lidar geometric registration residuals, point cloud overlap rates, satellite positioning solution quality factors, and inspection terminal attitude jitter, the reliability of positioning results in abnormal areas is assessed. When the reliability meets the requirements, the positioning results in abnormal areas are considered valid inspection results and recorded in the background inspection log. When the reliability is insufficient, the positioning results in abnormal areas are marked as pending review, triggering a local backtracking correction process to readjust the pose solution and spatial mapping results of the inspection terminal.

[0060] In obtaining the set of abnormal candidate regions Then, the center curve of the highway alignment for the currently inspected section is retrieved. And utilize abnormal candidate centroids By performing nearest neighbor projection matching with the route center curve, the mileage parameters corresponding to the anomaly candidate region are obtained: ; in, Mileage parameters The corresponding centerline coordinates of the route; Candidate centroids for anomalies The corresponding optimal mileage parameters; This indicates the search for optimal mileage parameters. The function is used to map spatial objects in the slope scene coordinate system back to the stationing system commonly used in highway engineering, so as to directly connect with inspection logs, maintenance work orders, and historical damage records.

[0061] To obtain the optimal mileage parameters Subsequently, the slope zoning rules, elevation zone division rules, and anomaly candidate boundaries were further combined. Generate structured localization results .in, It should include at least the coordinates of the center of the abnormal area, the spatial boundary range, the mileage marker, the slope zone, the elevation zone, the inspection time, and the data source identifier. In this way, slope problem areas that originally only showed local point cloud anomalies or near-field observation clues can be transformed into standardized spatial objects that can be directly archived, directly dispatched, and directly verified.

[0062] To enhance the consistency between the structured results and the management objects of highway slope engineering, the structured positioning results can be further refined based on the route driving direction, upper or lower slope attributes, protective structure type, and spatial adjacency relationships of adjacent anomalies. Supplementary labeling is performed. This step outputs a structured set of location results for the slope disease ledger and operation and maintenance platform, providing object-oriented input for the next step of credibility assessment.

[0063] Obtain structured localization results Subsequently, based on the geometric registration residual of the lidar Point cloud overlap rate GNSS solution quality factor and the attitude jitter of the inspection terminal To assess the reliability of the spatial location results for the current abnormal region, a reliability function is constructed: ; in, Abnormal area The credibility of the location; For Sigmoid mapping functions; For the registration residual term weighting coefficients; The weighting coefficient for the point cloud overlap rate term; For GNSS quality items, the weighting coefficients are used. The weighting coefficient for the attitude jitter term; For the calendar Point cloud overlap rate; Calculate the quality factor for GNSS; This represents the attitude jitter of the inspection terminal. This confidence function is used to measure whether the current positioning results have sufficient spatial reliability and engineering usability.

[0064] Era Point cloud overlap rate for: ; in, For the calendar After spatial transformation, the point cloud is at a distance threshold from the reference scene model or reference point cloud. The number of points that were successfully matched within the range; For the calendar The total number of valid points participating in the matching in the point cloud below; Used to indicate the degree of overlap between the current point cloud and the reference geometry.

[0065] When satisfied When this happens, the point is determined to be an effective overlapping point.

[0066] GNSS solution quality factor The expression is: ; in, To locate the solution state index, those skilled in the art preset its corresponding values; The number of satellites participating in the solution; This represents the normalized upper limit for the number of satellites. This is the accuracy attenuation factor; This is the normalized upper limit of the accuracy attenuation factor; This is the signal-to-noise ratio statistical value; This represents the upper limit of the normalized signal-to-noise ratio. , , and Let be the weighting coefficient for the GNSS quality sub-item, and satisfy: ; After normalization, the value is in the range of [0,1]. The larger the value, the higher the quality of GNSS solution.

[0067] In some alternative embodiments, epoch The following is a measurement of the attitude jitter of the inspection terminal. The calculation formula is: ; in, The number of epochs contained in the sliding time window, which is greater than or equal to 2; It is greater than or equal to M; For the calendar Quaternion of the attitude of the inspection terminal; For the calendar Quaternion of the attitude of the inspection terminal; This is a quaternion logarithmic mapping; This represents quaternion multiplication. Attitude jitter is used to represent the degree of attitude fluctuation of the inspection terminal during continuous inspection; the larger the value, the more unstable the attitude of the inspection terminal.

[0068] In some alternative embodiments, epoch The following is a measurement of the attitude jitter of the inspection terminal. It can also be obtained by weighted summation of the variances of roll, pitch, and yaw angles within a sliding time window: ; in, , and The roll angle, pitch angle, and yaw angle are respectively; Represents the variance function; , and These are the corresponding weighting coefficients.

[0069] In one implementation, to ensure that the above parameters have a unified dimension, the lidar geometric registration residual, point cloud overlap rate, GNSS solution quality factor, and attitude jitter can be normalized first, and then substituted into the confidence function for comprehensive evaluation.

[0070] when If the location result for the abnormal area is deemed unreliable, a local backtracking correction process is triggered. The local backtracking correction process returns to step 2, where the unified pose of the inspection terminal is resolved under conditions of expanding the time window or readjusting the joint optimization weights. Then, steps 3 through 5 are executed sequentially to obtain new structured localization results. Here, This is the reliability threshold. This closed-loop mechanism automatically prevents low-quality positioning results from directly entering the operational system when there are GNSS signal fluctuations, insufficient point cloud overlap, or drastic changes in the attitude of the inspection terminal.

[0071] The reliability threshold can be determined based on the statistical distribution of historical inspection results, the verification results of known accurate positioning samples, or the positioning accuracy requirements of the business system. It can also be set in different levels according to different inspection grades, slope types, and operating environments. In one implementation, the reliability threshold can be statistically determined using historically verified samples, ensuring that results exceeding this threshold meet preset positioning accuracy requirements. In another implementation, the reliability threshold can be set separately for high-risk slopes, ordinary slopes, or complex, obstructed scenarios.

[0072] when At that time, the structured positioning results will be... The data is entered into the slope inspection log and integrated with the maintenance work order system, disease verification system, or digital operation and maintenance platform. This forms a complete closed loop of "multi-source data acquisition—unified time synchronization—status propagation—joint registration—spatial mapping—anomaly candidate generation—mileage association—reliable verification—results storage," enabling multi-source spatiotemporal registration and positioning for highway slope inspection scenarios.

[0073] This invention proposes a collaborative construction mechanism for a multi-source unified time reference and a unified scene coordinate system for highway slope inspection scenarios. Instead of performing identification before positioning, it establishes a spatiotemporal integrated constraint chain during the data acquisition phase. It also proposes a joint registration and positioning mechanism that integrates inspection terminal pose, lidar geometric constraints, GNSS absolute positioning constraints, and slope alignment priors, enabling abnormal areas in slope inspection scenarios to be directly mapped to three-dimensional coordinates, station numbers, and slope zoning. Furthermore, it proposes a structured spatial representation mechanism for slope inspection results, transforming traditional image-level anomaly results into a unified object of "spatial location—slope zoning—mileage station number—elevation zone—reliability." Finally, it proposes a closed-loop correction mechanism for positioning reliability based on registration residuals, point cloud overlap, GNSS quality, and attitude jitter, enabling the system to possess degradeable, traceable, and verifiable engineering capabilities under complex operating conditions.

[0074] In one or more embodiments, a multi-source spatiotemporal registration and positioning system for highway slope inspection is provided, which can be implemented in software. The multi-source spatiotemporal registration and positioning system for highway slope inspection includes the following software modules: The multi-source synchronous sensing module is used to synchronously receive multi-source sensing data, which includes images from the inspection terminal's perspective, three-dimensional point clouds from lidar, inertial measurement information, satellite positioning information, and prior data of the slope scene. The predictive state correction module is used to calculate the continuous motion state of the inspection terminal in the slope scene based on inertial measurement information. It performs multi-source joint correction under the absolute positioning constraints, lidar geometric registration constraints and slope linear prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, to obtain the pose of the inspection terminal in the slope scene coordinate system. The near-field observation sector construction module is used to uniformly map the three-dimensional point cloud of the lidar to the slope scene coordinate system according to the calibration relationship between the inspection terminal and the lidar, and then construct the near-field observation sector at the current moment according to the current pose, field of view and effective observation range of the inspection terminal. The abnormal region identification module is used to divide the near-field observation sector coverage area into several candidate spatial units. Based on the local elevation difference change, point cloud density change and continuous observation attention value of each candidate spatial unit, the abnormal region is identified and its boundary range and spatial center position are extracted. The anomaly area credibility assessment module is used to match the spatial center location of the anomaly area with the route centerline of the current inspection section to obtain the corresponding mileage station information. Then, combined with the slope zoning rules and elevation zone division rules, the anomaly area is mapped into a standardized structured positioning result and its credibility is assessed.

[0075] It should be noted that each module in the multi-source spatiotemporal registration and positioning system for highway slope inspection in this embodiment corresponds one-to-one with each step in the multi-source spatiotemporal registration and positioning method for highway slope inspection in the above embodiment, and their specific implementation processes are the same, so they will not be repeated here.

[0076] The structure of the electronic device according to an embodiment of the present invention is described in detail below. The electronic device includes at least one processor, a memory, a user interface, and at least one network interface. The various components in the multi-source spatiotemporal registration and positioning system for highway slope inspection are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The user interface may include a display, keyboard, mouse, trackball, click wheel, buttons, a touchpad, or a touch screen, etc.

[0077] It is understood that the memory can be volatile memory or non-volatile memory, or both. The memory in this embodiment of the invention is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0078] In some embodiments, the multi-source spatiotemporal registration and positioning system for highway slope inspection provided in this invention can be implemented using a combination of hardware and software. For example, the system can be a processor in the form of a hardware decoding processor, programmed to execute the multi-source spatiotemporal registration and positioning method for highway slope inspection provided in this invention. For instance, the hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0079] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0080] As an example of the hardware implementation of the multi-source spatiotemporal registration and positioning system for highway slope inspection provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the multi-source spatiotemporal registration and positioning method for highway slope inspection provided in this embodiment of the invention.

[0081] The memory in this embodiment of the invention is used to store various types of data to support the operation of a multi-source spatiotemporal registration and positioning system for highway slope inspection, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operation on a multi-source spatiotemporal registration and positioning system for highway slope inspection, such as executable instructions. A program implementing the multi-source spatiotemporal registration and positioning method for highway slope inspection according to embodiments of the present invention can be included in the executable instructions.

[0082] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-source spatiotemporal registration and positioning method for highway slope inspection, characterized in that, include: Simultaneously receive multi-source sensing data, including images from the inspection terminal's perspective, 3D point clouds from LiDAR, inertial measurement information, satellite positioning information, and prior data of the slope scene; Based on inertial measurement information, the continuous motion state of the inspection terminal in the slope scene is calculated. Under the absolute positioning constraints, lidar geometric registration constraints and slope linear prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, multi-source joint correction is performed to obtain the pose of the inspection terminal in the slope scene coordinate system. The three-dimensional point cloud of the lidar is uniformly mapped to the slope scene coordinate system, and then the near-field observation sector at the current moment is constructed based on the current pose, field of view and effective observation range of the inspection terminal. Within the coverage area of ​​the near-field observation sector, several candidate spatial units are divided. Based on the local elevation difference changes, point cloud density changes, and continuous observation attention values ​​of the inspection terminal for each candidate spatial unit, abnormal areas are identified and their boundary range and spatial center location are extracted. The spatial center of the abnormal area is matched with the center line of the current inspection section to obtain the corresponding mileage station information. Then, combined with the slope zoning rules and elevation zone division rules, the abnormal area is mapped into a standardized structured positioning result and its credibility is evaluated. The process of calculating the continuous motion state of the inspection terminal in a slope scenario based on inertial measurement information is as follows: Define the state vector of the inspection terminal at each epoch. Based on the posterior state of the previous epoch and the current inertial measurement information, use the inertial propagation function to make prior predictions on the state of the inspection terminal. During inertial propagation, inertial measurement information is recursively updated, and smoothing constraints are applied to attitude changes in adjacent epochs to suppress the impact of short-term attitude changes caused by slope uphill and downhill operations, vegetation obstruction and detour, and local sudden stop observation on attitude prediction, so as to obtain the predicted state of the inspection terminal. The objective function corresponding to the pose of the inspection terminal in the slope scene coordinate system is the optimal pose estimate corresponding to the weighted sum of the absolute positioning constraint, the lidar geometric registration constraint, and the slope alignment prior constraint. Among them, the absolute positioning constraint is the position residual of the inspection terminal; the lidar geometric registration constraint is the lidar geometric registration residual; and the slope alignment prior constraint is the slope prior consistency residual.

2. The multi-source spatiotemporal registration and positioning method for highway slope inspection as described in claim 1, characterized in that, Candidate spatial cells are generated according to the local mesh, voxel cells or slope slices of the slope surface.

3. The multi-source spatiotemporal registration and positioning method for highway slope inspection as described in claim 1, characterized in that, The continuous observation attention value of the inspection terminal is generated by combining the dwell time, the number of times the line of sight is repeatedly pointed, or the edge trigger record.

4. The multi-source spatiotemporal registration and positioning method for highway slope inspection as described in claim 1, characterized in that, Based on standardized structured positioning results, and combined with lidar geometric registration residuals, point cloud overlap rate, satellite positioning solution quality factor, and inspection terminal attitude jitter, the reliability of positioning results in abnormal areas is assessed.

5. The multi-source spatiotemporal registration and positioning method for highway slope inspection as described in claim 4, characterized in that, When the credibility requirement is met, the location results of the abnormal area are taken as valid inspection results and written into the background inspection log. When the credibility is insufficient, the location results of the abnormal area are marked as pending review, triggering a local backtracking correction process to readjust the pose solution and spatial mapping results of the inspection terminal.

6. A multi-source spatiotemporal registration and positioning system for highway slope inspection, characterized in that, The multi-source spatiotemporal registration and positioning method for highway slope inspection based on any one of claims 1-5 includes: The multi-source synchronous sensing module is used to synchronously receive multi-source sensing data, which includes images from the inspection terminal's perspective, three-dimensional point clouds from lidar, inertial measurement information, satellite positioning information, and prior data of the slope scene. The predictive state correction module is used to calculate the continuous motion state of the inspection terminal in the slope scene based on inertial measurement information. It performs multi-source joint correction under the absolute positioning constraints, lidar geometric registration constraints and slope linear prior constraints corresponding to satellite positioning information, lidar 3D point cloud and slope scene prior data, to obtain the pose of the inspection terminal in the slope scene coordinate system. The near-field observation sector construction module is used to uniformly map the three-dimensional point cloud of the lidar to the slope scene coordinate system, and then construct the near-field observation sector at the current moment based on the current pose, field of view and effective observation range of the inspection terminal. The abnormal region identification module is used to divide the near-field observation sector coverage area into several candidate spatial units. Based on the local elevation difference change, point cloud density change and continuous observation attention value of each candidate spatial unit, the abnormal region is identified and its boundary range and spatial center position are extracted. The anomaly area credibility assessment module is used to match the spatial center location of the anomaly area with the route centerline of the current inspection section to obtain the corresponding mileage station information. Then, combined with the slope zoning rules and elevation zone division rules, the anomaly area is mapped into a standardized structured positioning result and its credibility is assessed.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multi-source spatiotemporal registration and positioning method for highway slope inspection as described in any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multi-source spatiotemporal registration and positioning method for highway slope inspection as described in any one of claims 1-5.

Images