Highway inspection target positioning method based on multi-source data fusion

Abstract

The invention provides a road inspection target positioning method based on multi-source data fusion, and relates to the field of road daily inspection and infrastructure performance monitoring, and the method comprises the following steps: obtaining multi-source data based on a preset multi-source sensor; through deep coupling of satellite positioning carrier double-difference observation and inertial navigation solution of an inertial sensor, error correction is carried out in combination with Kalman filtering recursion, and fusion navigation data is determined; according to the fused navigation data, carrying out fusion processing on the two-dimensional visual observation data and the three-dimensional space point cloud data to obtain a forward view depth map; determining the national geodetic coordinates of the disease by combining a depth value back calculation method and utilizing a forward view depth map; and obtaining a starting kilometer pile number in a road inspection process, and determining a mileage pile number of the disease by combining the national geodetic coordinate of the disease and utilizing a Gaussian positive algorithm. According to the invention, centimeter-level geographic coordinate positions of diseases and corresponding mileage stake numbers can be output.

Description

Technical Field

[0001] This invention relates to the field of routine highway inspection and infrastructure performance monitoring, and more specifically, to a method for locating highway inspection targets based on multi-source data fusion. Background Technology

[0002] Currently, mainstream road inspection equipment mainly relies on high-precision vehicle positioning methods, using algorithms to calculate the absolute position of the inspection target. However, its positioning accuracy is only at the meter level, which cannot meet the needs of refined management in many cases. In addition, during actual operations, manual intervention measures such as stake adjustment are often required to correct positioning errors. This not only increases the workload but also makes the data inaccurate due to human factors.

[0003] Specifically, current technologies face several major challenges: Existing inspection equipment often employs a single GNSS positioning or low-precision inertial navigation system, making it difficult to provide centimeter-level accurate positioning information. This poses a challenge for applications such as defect detection, asset management, and long-term performance monitoring. Traditional methods often require manual intervention for stake calibration to ensure positioning accuracy. This process is time-consuming and susceptible to human error, reducing inspection efficiency and reliability. Although two-dimensional visual observation data (such as camera images) and three-dimensional spatial point cloud data (such as LiDAR scans) can provide rich environmental information, current technologies still have significant shortcomings in efficiently fusing these different dimensions of data and utilizing them for precise positioning.

[0004] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0005] In view of this, the present invention provides a method for locating highway inspection targets based on multi-source data fusion to solve the aforementioned problems.

[0006] To solve the above problems, the specific technical solution adopted by the present invention is as follows:

[0007] A method for locating highway inspection targets based on multi-source data fusion, comprising the following steps:

[0008] S1. Based on preset multi-source sensors, acquire multi-source data including satellite positioning data, inertial sensor data, two-dimensional visual observation data and three-dimensional spatial point cloud data during highway inspection.

[0009] S2. Based on the acquired satellite positioning data and inertial sensor data, the fused navigation data is determined by deeply coupling the satellite positioning carrier double-difference observation with the inertial navigation solution of the inertial sensor, and by combining the Kalman filter recursive error correction.

[0010] S3. Based on the fused navigation data, the two-dimensional visual observation data and the three-dimensional spatial point cloud data are fused to obtain the front view image depth map; and combined with the inverse calculation method of depth value, the national geodetic coordinates of the disease are determined using the front view image depth map.

[0011] S4. Obtain the starting kilometer marker number during the highway inspection process, and combine it with the national geodetic coordinates of the defects to determine the mileage marker number of the defects using the Gaussian forward algorithm.

[0012] Preferably, the determination of fused navigation data based on the acquired satellite positioning data and inertial sensor data, through deep coupling of satellite positioning carrier double-difference observation and inertial navigation calculation of inertial sensors, and combined with Kalman filter recursive error correction, includes the following steps:

[0013] S21. Based on the acquired satellite positioning data, the pseudorange double difference value and the carrier phase double difference value are calculated respectively using the satellite positioning carrier observation model and the carrier double difference equation.

[0014] S22. Based on inertial sensor data and using the INS mechanical arrangement equations to perform inertial navigation calculations, the current state vector containing position, velocity and attitude estimates is obtained at the current moment. Combined with the state transition matrix, the predicted state vector and covariance matrix for the next moment are predicted.

[0015] S23. Based on the current position, velocity, and attitude estimates, and combined with the satellite ephemeris and base station location, calculate the double-difference geometric distance predicted by the INS.

[0016] S24. Subtract the pseudorange double difference value and the carrier phase double difference value from the double difference geometric distance respectively to form the double difference observation residual. Combine the residual with the pre-configured measurement equation to calculate the Kalman gain and form the gain matrix.

[0017] S25. Using the gain matrix, the predicted state vector is corrected to obtain the optimal estimates of the inertial component errors and navigation parameter errors; the inertial component errors include gyroscope drift and accelerometer bias; the navigation parameter errors include attitude angle errors, velocity errors, and position errors.

[0018] S26. Using the optimal estimates of inertial component errors and navigation parameter errors, perform dual feedback correction on the inertial sensor data to obtain fused navigation data.

[0019] Preferably, the step of using the optimal estimates of inertial component errors and navigation parameter errors to perform dual feedback correction on inertial sensor data to obtain fused navigation data includes the following steps:

[0020] S251. Convert the attitude error into an attitude correction value through coordinate transformation, and update the attitude matrix of the INS to obtain the updated attitude data.

[0021] S252. The velocity error and position error are superimposed on the velocity and position calculation results of the INS to eliminate the cumulative deviation of navigation parameters and obtain updated velocity and position data.

[0022] S253. Integrate the updated attitude data with the updated velocity and position data to obtain fused navigation data.

[0023] Preferably, the step of fusing two-dimensional visual observation data and three-dimensional spatial point cloud data based on fused navigation data to obtain a front view image depth map, and then using the front view image depth map to determine the national geodetic coordinates of the disease using a depth value inverse calculation method, includes the following steps:

[0024] S31. Perform spatiotemporal joint calibration on the camera that acquires two-dimensional visual observation data and the laser scanner that acquires three-dimensional spatial point cloud data, and solve for the camera extrinsic parameters.

[0025] S32. Based on the shooting angle and field of view of the two-dimensional visual observation data, the point cloud data corresponding to the two-dimensional visual observation data is selected from the 3D laser point cloud to obtain the selected point cloud data.

[0026] S33. By transforming the coordinates, the filtered point cloud data is converted to the pixel coordinate system of the two-dimensional visual observation data to obtain the depth map of the front view image;

[0027] S34. Based on the two-dimensional visual observation data, the pixel coordinates of the disease are determined by the YOLOv10 algorithm. The calculated depth value is determined by combining the fused navigation data and the depth map of the front view image. Based on the pixel coordinates of the disease and the outer side of the camera, the geodetic coordinates of the disease are calculated.

[0028] Preferably, the step of transforming the filtered point cloud data to the pixel coordinate system of the two-dimensional visual observation data through coordinate transformation to obtain the front view image depth map includes the following steps:

[0029] S331. Through geographic coordinate transformation, the coordinates of each filtered point cloud data in the preset world geodetic coordinate system are converted to the coordinates in the local horizontal coordinate system.

[0030] S332. Based on the attitude angle corresponding to the two-dimensional visual observation data at the shooting time, convert the coordinates of each filtered point cloud data in the local horizontal coordinate system to the coordinates in the IMU coordinate system.

[0031] S333. Transform the coordinates of each filtered point cloud data in the IMU coordinate system to the image space rectangular coordinate system, and calculate the coordinates of each filtered point cloud data in the image plane coordinate system using the collinearity equation.

[0032] S334. Convert the coordinates of each filtered point cloud data in the image plane coordinate system to the pixel coordinate system to achieve the fusion of the two-dimensional visual observation data and the filtered point cloud data, and obtain the front view image depth map.

[0033] Preferably, the step of determining the pixel coordinates of the lesion based on two-dimensional visual observation data using the YOLOv10 algorithm, determining the calculated depth value by combining fused navigation data and the depth map of the foreground image, and calculating the geodetic coordinates of the lesion based on the pixel coordinates of the lesion and the outer edge of the camera includes the following steps:

[0034] S341. Using a deep neural network structure, road scene segmentation is performed on two-dimensional visual observation data to obtain the road surface part and the non-road surface part;

[0035] S342. Based on the road surface, the YOLOv10 algorithm is used to identify and locate targets in the region of interest on the road surface, and the detection results located in non-road surface areas are removed. Combined with the preset scene constraints and the target category information, the small box information is merged, and the pixel coordinates of the defects in the two-dimensional visual observation data are output.

[0036] S343. Based on the pixel coordinates of the lesion and the fused navigation data, read the depth value at the corresponding position on the front view image depth map. Based on the pixel coordinates of the lesion and the outside of the camera, determine the geodetic coordinates of the lesion through inverse calculation of the depth value.

[0037] Preferably, the step of reading depth values ​​at the corresponding positions on the front view image depth map based on the defect pixel coordinates and fused navigation data, and determining the geodetic coordinates of the defect through inverse calculation of the depth values ​​based on the defect pixel coordinates and the outer edge of the camera includes the following steps:

[0038] S3431. Based on the coordinates of the defective pixels, read the depth value at the corresponding position in the depth map of the front view image;

[0039] S3432. Obtain the camera's intrinsic parameters and convert the pixel coordinates of the lesion into image plane coordinates;

[0040] S3433. Based on the image plane coordinates, and according to the depth value and the camera focal length, the three-dimensional coordinates of the disease in the image space rectangular coordinate system are derived using the principle of similar triangles.

[0041] S3434. Based on camera extrinsic parameters, convert the three-dimensional coordinates in the image space rectangular coordinate system into the disease coordinates in the IMU coordinate system;

[0042] S3435. Using the transformation matrix corresponding to the IMU attitude angle, convert the defect coordinates in the IMU coordinate system to the defect coordinates in the local horizontal coordinate system.

[0043] S3436. Convert the national geodetic coordinates of the two-dimensional visual observation data shooting point into Gaussian plane coordinates through Gaussian forward calculation, and superimpose the disease coordinates in the local horizontal coordinate system onto the Gaussian coordinates of the shooting point to obtain the Gaussian plane coordinates and elevation of the disease.

[0044] S3437. Convert the Gaussian plane coordinates to national geodetic coordinates through Gaussian inverse calculation to obtain the national geodetic coordinates of the disease.

[0045] Preferably, the step of obtaining the starting kilometer marker number during highway inspection and determining the mileage marker number of the defect using the Gaussian forward algorithm, in conjunction with the national geodetic coordinates of the defect, includes the following steps:

[0046] S41. Obtain the starting kilometer marker number and the national geodetic coordinates of the starting kilometer marker number during the highway inspection process.

[0047] S42. Construct an initial base map containing the latitude and longitude coordinates of the road network station numbers and lane lines. Combine this with the national geodetic coordinates of the starting kilometer station number. On the initial base map, draw perpendicular lines from the starting kilometer station number and the defect to the center line of the highway lane, respectively, to obtain the first intersection point and the second intersection point.

[0048] S43. Calculate the national geodetic coordinates of the first and second intersection points in the national geodetic coordinate system using the Gaussian forward calculation formula;

[0049] S44. Based on the idea of ​​differentiation, and according to the national geodetic coordinates of the first and second intersection points, calculate the curve distance between the first and second intersection points, and determine the mileage station number of the defect according to the starting kilometer station number.

[0050] Preferably, the method of calculating the curve distance between the first and second intersection points based on the idea of ​​differentiation and according to the national geodetic coordinates of the first and second intersection points in the national geodetic coordinate system, and determining the mileage station of the defect according to the starting kilometer marker number, includes the following steps:

[0051] S441. Divide the curve between the first intersection point and the second intersection point into several sub-intervals;

[0052] S442. For each subinterval, use the geodetic coordinates of its endpoints or representative points within the interval to calculate the approximate local arc length of that subinterval.

[0053] S443. Sum the approximate local arc length values ​​of all sub-intervals to obtain the curve distance between the first intersection point and the second intersection point, and determine the mileage station number corresponding to the location of the defect based on the starting kilometer station number.

[0054] Preferably, the formula for calculating the approximate local arc length of each sub-interval using the geodetic coordinates of its endpoints or representative points within the interval is as follows:

[0055] ;

[0056] In the formula, This represents the approximate local arc length of the i-th subinterval. This represents the coordinate increment along the x-axis. This represents the coordinate increment along the y-axis. Indicates the curve at x i The derivative at point, This represents the x-coordinate of the point.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. Based on the high-precision positioning needs of disease tracing, multi-period data comparison and asset inventory, this invention has developed an automatic staking and high-precision positioning algorithm for diseases based on multi-source data fusion of GNSS, inertial navigation and lidar, which can output the centimeter-level geographic coordinates of diseases and the corresponding mileage station number.

[0059] 2. This invention performs algorithmic matching of 2D images and 3D laser point cloud multidimensional information. Through the automatic pile correction algorithm of the inspection system, it automatically connects the GNSS / INS fusion high-precision positioning technology with the mileage pile data to achieve automatic matching and correction between the inspection target and the target pile number.

[0060] 3. This invention uses a real-time Kalman filter method to acquire navigation and positioning data, and performs multi-source data fusion of navigation and positioning data, point cloud data, and image data to support the creation of a three-dimensional digital road scene; furthermore, it develops an automatic mileage marker calibration algorithm for an inspection system based on road network electronic fences, which for the first time achieves centimeter-level positioning of inspection targets and real-time matching of mileage markers. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0062] Figure 1This is a flowchart according to an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of a loosely combined GNSS / IMU structure according to an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of a tightly integrated GNSS / IMU structure according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the vehicle trajectory in experimental scenario-01 according to an embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the vehicle in experimental scenario-02 according to an embodiment of the present invention;

[0067] Figure 6 This is one of the comparative diagrams of GNSS and GNSS / INS according to an embodiment of the present invention;

[0068] Figure 7 This is the second comparative diagram of GNSS and GNSS / INS according to an embodiment of the present invention;

[0069] Figure 8 This is one of the statistical comparison diagrams of satellite obstruction status of data-01 with GNSS and GNSS / INS calculation results according to an embodiment of the present invention;

[0070] Figure 9 This is the second schematic diagram showing the statistical comparison between satellite obstruction status of data-01 and GNSS / INS calculation results in an embodiment of the present invention.

[0071] Figure 10 This is a schematic diagram of the front view image measurement principle according to an embodiment of the present invention;

[0072] Figure 11 These are front view images and depth maps according to embodiments of the present invention;

[0073] Figure 12 This is a front view of a building according to an embodiment of the present invention;

[0074] Figure 13 This is a rendering of the building's front view scene fusion according to an embodiment of the present invention;

[0075] Figure 14 This is a scene diagram of a building corridor according to an embodiment of the present invention;

[0076] Figure 15 This is a rendering of the architectural corridor scene fusion according to an embodiment of the present invention;

[0077] Figure 16This is an open road scene diagram according to an embodiment of the present invention;

[0078] Figure 17 This is a fusion effect diagram of an open road scene map according to an embodiment of the present invention;

[0079] Figure 18 This is a mileage station calculation diagram according to an embodiment of the present invention. Detailed Implementation

[0080] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0081] According to an embodiment of the present invention, a method for locating highway inspection targets based on multi-source data fusion is provided.

[0082] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to a first embodiment of the present invention, a method for locating highway inspection targets based on multi-source data fusion is provided, the method comprising the following steps:

[0083] S1. Based on preset multi-source sensors, acquire multi-source data during highway inspection, including satellite positioning data (GNSS data), inertial sensor data (IMU data), two-dimensional visual observation data (2D image data), and three-dimensional spatial point cloud data (3D laser point cloud data);

[0084] Specifically, GNSS data is used to provide a global positioning benchmark and achieve basic positioning; IMU data completes velocity, position and attitude updates through strapdown inertial navigation update algorithms; 2D image data is used to record visual information of the inspected target, and 3D laser point cloud data is used to capture the geometric features of the inspected target.

[0085] 3D road data can accurately collect and record the geometric features and key elements of the road surface and its surrounding environment, comprehensively reflecting the spatial topology and attribute information of the road environment, providing a rich and accurate foundation for road condition monitoring and analysis. The 3D road data, through navigation and positioning information provided by a POS attitude determination and positioning system composed of GNSS / INS, achieves a high-precision spatiotemporal positioning reference. GNSS provides global positioning information to ensure accurate positioning in open environments, while INS provides dynamic state estimation through an inertial measurement unit. Especially when GNSS signals are blocked or interfered with, INS can self-compensate using the measurement results of accelerometers and gyroscopes, thus maintaining continuous and stable positioning accuracy. The combination of these two technologies ensures high-precision spatiotemporal consistency of 3D road data in complex environments.

[0086] S2. Based on the acquired satellite positioning data and inertial sensor data, the fused navigation data is determined by deeply coupling the satellite positioning carrier double-difference observation with the inertial navigation solution of the inertial sensor, and by combining the Kalman filter recursive error correction.

[0087] In a preferred embodiment, the determination of fused navigation data based on the acquired satellite positioning data and inertial sensor data, through deep coupling of satellite positioning carrier double-difference observation and inertial navigation calculation of inertial sensors, and combined with Kalman filter recursive error correction, includes the following steps:

[0088] S21. Based on the acquired satellite positioning data, the pseudorange double difference value and the carrier phase double difference value are calculated respectively using the satellite positioning carrier observation model and the carrier double difference equation.

[0089] It should be noted that the satellite positioning carrier observation model, i.e., the GNSS positioning model, is a real-time kinematic presentation (RTK) positioning system. RTK is a real-time dynamic carrier phase differential technique. The implementation of RTK relies on GNSS. Among high-precision positioning methods, RTK-based algorithms are preferred because they can provide static millimeter-level and dynamic centimeter-level services. In urban environments, due to limitations in observation conditions and hardware costs, RTK technology has become the preferred choice for high-precision, real-time positioning.

[0090] The expression for the GNSS carrier observation model is shown below:

[0091] ;

[0092] In the formula, The pseudorange (m) is the observed value at each frequency; This is a geometric distance measure, representing the distance between the satellite's current position and the initial estimated position. The speed of light; and These are the receiver and satellite clock differences relative to the reference time system, respectively; The ionospheric delay at the i-th frequency; For tropospheric delay; and These are the receiver hardware delay and satellite clock hardware delay for pseudorange observations, respectively. This refers to the multipath delay, while This refers to the participation error in modeling other aspects.

[0093] GNSS carrier differential positioning, also known as RTK positioning technology, uses a carrier double-difference equation for solution to eliminate hardware delay and clock bias, as shown in the following equation:

[0094] ;

[0095] In the formula, Tr is the reference target temperature, Tb is the background temperature, and TG and TB represent the equivalent radiation temperatures corresponding to the G-band and B-band, respectively; N rb,G、 N rb,B These are the noise terms for the G-band and B-band, respectively, used to characterize the random error of the system. , , These are the delayed projection parameters for the ionosphere, the reference station's troposphere, and the rover's troposphere, respectively. The identity matrix is ​​used to construct the coefficient relationships of the equations, ensuring parameter dimension matching and computational rationality. It is a fundamental matrix element in linear observation models. The wavelength of B-band electromagnetic waves is related to λ. G The (G-band wavelength) corresponds to the core physical parameter that characterizes the radiation and propagation characteristics of this band and is used to distinguish the observation characteristics of different bands.

[0096] Considering the practical application of short baseline estimation, it is unnecessary to estimate spatially correlated parameters such as the ionosphere and troposphere. These parameters are eliminated in the double-difference equation, resulting in the equation shown below:

[0097] ;

[0098] In the formula, For gradient operators, P rb,G and , These are the observed feature quantities of the target region in the G channel and the B channel, respectively. This represents the theoretical reflectance for the corresponding G channel and B channel. For incident radiation energy, For reference reflection model parameters, , For channel system bias terms; X is the weighting coefficient, and E is the energy function; , These are the correction and compensation parameters to be solved.

[0099] S22. Based on inertial sensor data and using the INS mechanical arrangement equations to perform inertial navigation calculations, the current state vector containing position, velocity and attitude estimates is obtained at the current moment. Combined with the state transition matrix, the predicted state vector and covariance matrix for the next moment are predicted.

[0100] Mechanical orchestration is performed in the navigation coordinate system. Before mechanical orchestration of the strapdown inertial navigation system, it is necessary to acquire the position, velocity, and attitude data of the inertial sensors, and initialize the quaternions of the navigation system to ground and ground-fixed system, the quaternions of the carrier X-axis to the navigation system, the equivalent cosine matrix of the attitude angles, and the velocity of the previous epoch. Specifically, this includes:

[0101] 1. Speed ​​update: Based on the speed differential equation in the INS mechanical arrangement, the speed update formula can be obtained by integrating over the interval time:

[0102] ;

[0103] 2. Position update, based on the four elements of position. The system then updates the latitude and longitude, thus completing the process of updating the latitude and longitude positions.

[0104] ;

[0105] in, and They can be expressed in the following forms:

[0106] ;

[0107] ;

[0108] ;

[0109] In the formula, Let be the quaternion for the transformation from the navigation system (n-1) to the Earth system (e-1) at time k. Let be the quaternion for the transformation from the navigation frame (n-1) to the Earth frame (e-1) at time k-1. Let be the quaternion representing the transformation from the navigation frame at time k-1 to the navigation frame at time k. Let be the quaternion for the transformation from the navigation frame (n-frame) to the Earth frame (e-frame) at time k. Let be the quaternion for the transformation from the navigation frame at time k-1 to the navigation frame at time k. Let e ​​be the rotation vector of the e-system between adjacent epochs.

[0110] The velocity at intermediate moments is obtained by interpolating the velocity updated at the current moment and the velocity at the previous moment using the following formula.

[0111] ;

[0112] In the formula, The velocity at the midpoint of the time interval. The velocity at the previous moment, The speed is the speed updated at the current moment.

[0113] The precision and latitude can be obtained by parsing the position quaternion, and the height can be updated separately using the following formula:

[0114] ;

[0115] In the formula, Let the height be at time k. The height at time k-1. Let be the vertical velocity at time k-1. The time interval is from time k-1 to time k.

[0116] 3. Pose Update: The pose four-element update algorithm is shown in the following formula:

[0117] ;

[0118] In the formula, the four elements of the b system between time k and time k-1 The calculation formula is:

[0119] ;

[0120] In the formula, For the four elements of the b-series between time k and time k-1, Let be the transformation quaternion from the carrier coordinate system (b system) at time k-1 to the navigation coordinate system (n system) at time k-1. Let be the transformation quaternion from the carrier coordinate system (b system) at time k to the carrier coordinate system (b system) at time k-1. Let be the transformation quaternion from the carrier coordinate system (b-frame) at time k to the navigation coordinate system (n-frame) at time k. For the quaternion updated in the n-series, For the rotation vector in the carrier coordinate system, in engineering applications, the approximate expression of the equivalent rotation vector differential equation is:

[0121] ;

[0122] Integrating the above equivalent rotating vector differential equation yields the following expression:

[0123] ;

[0124] In the formula, To represent the rate of change of the attitude error angle, Let i be the angular velocity vector of the inertial frame (i) relative to the carrier frame (b) in the carrier coordinate system (b). The rotation vector in the carrier coordinate system. Let be the change in attitude angle at time k. Let be the change in attitude angle at time k-1. The second term represents the second-order conic effect correction term.

[0125] The quaternion of the n-series update can be calculated using the following formula:

[0126] ;

[0127] In the formula, For the quaternion updated in the n-series, Let be the rotation vector of the n-system at time k relative to time k-1.

[0128] The position at an intermediate time step can be calculated by interpolating the updated position at the current time step with the position at the previous time step. Height interpolation can be performed using the following formula:

[0129] ;

[0130] In the formula, This represents the position at the midpoint of the time interval. Let the height be at time k. The height is at time k-1.

[0131] Corresponding from arrive The quaternion representing the change in position at a given time can be calculated using the following formula:

[0132] ;

[0133] In the formula, for arrive Quaternions whose position changes over time. To represent the transformation quaternion from the navigation frame (n frame) to the Earth frame (e frame) at time k-1, Let q be the quaternion representing the transformation from the navigation frame (n frame) to the Earth frame (e frame) at time k.

[0134] Since the rotation vector can be calculated using the above formula, position interpolation can be calculated using the following formula:

[0135] ;

[0136] In the formula, This is the quaternion corresponding to 0.5δθ. Let q be the quaternion representing the transition from the navigation system (n-frame) to the Earth system (e-frame) at the midpoint between k-1 and k (i.e., (k-1 / 2) time).

[0137] Due to numerical errors, the calculated This often violates the constraints of normalization, so it needs to be normalized, as shown in the following formula:

[0138] ;

[0139] In the formula, e represents the normalization error in the quaternion. This is the transformation quaternion from the carrier coordinate system (b-frame) to the navigation coordinate system (n-frame). The normalization error for quaternions, This is the transpose operation of a matrix.

[0140] 4. INS error equation; INS error equation identification based on the C-system. The angular model is established as shown in the following formula:

[0141] ;

[0142] In the formula, Let be the second derivative of the position error in the navigation frame (n-frame). Let $\mathbf{n}$ be the angular velocity vector of the Earth system (e system) relative to the navigation system (n system) in the navigation frame (n frame). A general identifier for errors. The position vector in the navigation frame (n-frame). The velocity vector in the navigation frame (n-frame) is... The velocity error vector in the navigation frame (n-frame). Let n be the gravitational acceleration vector in the navigation frame (n-frame). Let i be the angular velocity vector of the inertial frame (i) relative to the Earth frame (e) in the navigation frame (n-frame). This is the specific force vector in the navigation frame (n-frame). The specific force vector in the load system (system b) is the force vector. The attitude error angle vector, Let be the direction cosine matrix from the carrier system (b-frame) to the navigation system (n-frame). Let i be the angular velocity vector of the inertial frame (i) relative to the Earth frame (e) in the navigation frame (n-frame). The measurement error of the angular velocity of the inertial frame (i) relative to the carrier frame (b) is given by the carrier frame (b). This represents the rate of change of the attitude error angle.

[0143] S23. Based on the current position, velocity, and attitude estimates, and combined with the satellite ephemeris and base station location, calculate the double-difference geometric distance predicted by the INS.

[0144] Specifically, when calculating the double-difference geometric distance predicted by INS, the carrier position and the known position of the base station calculated by INS are both converted into a geocentric coordinate system consistent with the satellite ephemeris. The satellite position is calculated using the satellite ephemeris, and the geometric distances from the base station to the satellite and from the carrier to the satellite are calculated separately. A reference satellite is selected, and the distance differences between the carrier and the two satellites, and between the base station and the two satellites, are calculated. The difference between these two differences is the double-difference geometric distance predicted by INS.

[0145] S24. Subtract the pseudorange double difference value and the carrier phase double difference value from the double difference geometric distance respectively to form the double difference observation residual. Combine the residual with the pre-configured measurement equation to calculate the Kalman gain and form the gain matrix.

[0146] Specifically, the calculation of Kalman gain, in conjunction with the pre-configured measurement equations, includes: first, determining the measurement equations, using the GNSS double-difference observation residuals as the measurement values, the measurement matrix being the partial derivative of the residuals with respect to the state (position / velocity / attitude error), then determining the covariance of the measurement noise, and finally substituting the predicted state covariance, combined with the measurement matrix and the measurement noise covariance, into the Kalman gain calculation formula to obtain the gain matrix.

[0147] S25. Using the gain matrix, the predicted state vector is corrected to obtain the optimal estimates of the inertial component errors and navigation parameter errors; the inertial component errors include gyroscope drift and accelerometer bias; the navigation parameter errors include attitude angle errors, velocity errors, and position errors.

[0148] In a preferred embodiment, the step of using the optimal estimates of inertial component errors and navigation parameter errors to perform dual feedback correction on inertial sensor data to obtain fused navigation data includes the following steps:

[0149] S251. Convert the attitude error into an attitude correction value through coordinate transformation, and update the attitude matrix of the INS to obtain the updated attitude data.

[0150] S252. The velocity error and position error are superimposed on the velocity and position calculation results of the INS to eliminate the cumulative deviation of navigation parameters and obtain updated velocity and position data.

[0151] S253. Integrate the updated attitude data with the updated velocity and position data to obtain fused navigation data.

[0152] S26. Using the optimal estimates of inertial component errors and navigation parameter errors, perform dual feedback correction on the inertial sensor data to obtain fused navigation data.

[0153] It should be noted that, from an algorithmic perspective, GNSS / INS integrated navigation can be divided into two modes based on different observation data and assistance methods: loose combination mode and tight combination mode. The principles of loose combination mode and tight combination mode are as follows: Figure 2-3 As shown, GNSS output information and inertial navigation system (INS) information are fused. The difference between the attitude and position data obtained from the two methods is used as the measurement output. The Kalman filter is then used to obtain the inertial component errors and navigation parameter errors, followed by feedback correction. In the GNSS / INS loose combination mode, the process of obtaining inertial component errors and navigation parameter errors through the Kalman filter consists of two stages: "time update - measurement update". First, based on the INS mechanical orchestration equations and combined with the system state transition matrix, the state vector and covariance matrix for the next moment are predicted. The state vector includes attitude, velocity, and position errors, as well as gyroscope / accelerometer drift, reflecting the dynamic propagation law of errors over time. Then, the difference between the attitude, position, and velocity data output from GNSS and INS is used as an observation, substituted into the measurement equations to calculate the Kalman gain. The predicted state vector is corrected through the gain matrix, ultimately obtaining the optimal estimates of inertial component errors and navigation parameter errors. Inertial component errors include gyroscope drift and accelerometer bias, while navigation parameter errors include attitude angle errors, velocity errors, and position errors.

[0154] Feedback correction is achieved through reverse error compensation: on the one hand, the attitude error is transformed into an attitude correction value through coordinate transformation, and the attitude matrix of the INS is updated. At the same time, the velocity error and position error are directly superimposed on the velocity and position calculation results of the INS to eliminate the cumulative deviation of navigation parameters. On the other hand, the estimated gyroscope drift and accelerometer bias are fed back to the sensor data preprocessing module of the INS to subtract the drift error from the original IMU measurement values, which include angular velocity, specific force, etc., to ensure that the subsequent inertial navigation calculation is based on the corrected sensor data, reduce the accumulation of error propagation, and improve the stability and positioning accuracy of the integrated navigation system.

[0155] In addition, to verify the accuracy of the GNSS / IMU integrated navigation, two sets of experimental data were collected using a vehicle-mounted mobile measurement system. The driving speed during the experiment was approximately 30 km / h. Experimental scenario-01 was collected on May 17, 2023, outside Shandong University of Science and Technology in Qingdao, with a trajectory length of approximately 7800m. The trajectory is as follows... Figure 4As shown, the road has three lanes, and the observation environment is relatively good. Experimental scenario-02 was collected on November 29, 2023, inside and outside Shandong University of Science and Technology in Qingdao, with a trajectory length of approximately 6300m. The trajectory is as follows... Figure 5 As shown, the roads on campus are narrow and densely shaded by trees on both sides, resulting in more severe occlusion conditions than in experimental scenario-01. The tightly coupled solution result from the IE software is the true pose reference value for the POS system.

[0156] like Figure 6-7 This chart presents a statistical comparison of satellite obstruction conditions with GNSS and GNSS / INS solution results. The chart shows the dynamic changes in GDOP, HDOP, PDOP, and VDOP. Overall, the DOP value varies considerably, especially GDOP (blue curve), which fluctuates significantly in certain time periods, indicating poor satellite geometric distribution. HDOP (red curve) and VDOP (green curve) values ​​are generally low, indicating relatively good positioning accuracy in both horizontal and vertical directions. However, occasional large fluctuations are due to signal obstruction. The number of satellites remains between 8 and 10 for most of the time, indicating a relatively wide field of view for the GNSS receiver during this period, allowing for the reception of sufficient satellite signals. However, the number of satellites decreases at epoch 268500, mainly due to environmental obstruction. The Ratio value typically indicates the success of the ambiguity fixation solution and the reliability of the positioning result. The chart shows that between epochs 267600 and 268800, the Ratio value fluctuates frequently and even becomes low, indicating unreliable GNSS positioning results. In this environment, GNSS alone exhibits significant error fluctuations across multiple time periods, demonstrating strong instability. This is primarily a result of experimental variations leading to GNSS satellite signal quality issues. GNSS positioning errors remain within 0.5 meters for the vast majority of the time, but exhibit large fluctuations in the epochs around 267,600-268,000 and 269,400-270,000. These fluctuations are mainly caused by deterioration in satellite signal quality, worsening geometric distribution, or multipath effects. In contrast, the advantages of GNSS / INS fusion are particularly evident, especially when the error fluctuates drastically or is interrupted in the epoch range of 269,400 to 270,000. The combined navigation error is generally controlled within 0.2 meters, meeting lane-level accuracy requirements.

[0157] like Figure 8-9This paper presents a statistical comparison of satellite obstruction conditions in Data-01 with GNSS and GNSS / INS solution results. Compared to Data-01, Data-02 exhibits more severe GNSS obstruction, with fewer satellites observed at certain time intervals. The decrease in satellite count at epoch 286800 is more significant than the corresponding decrease in Data-01. The frequency and amplitude of Ratio value fluctuations in Data-02 are greater than in Data-02 at certain time intervals, resulting in less stable GNSS positioning results in Data-02. Significant fluctuations are also observed in the epoch range of 287400-288000 in Data-2. Individual GNSS errors fluctuate across multiple time periods, demonstrating a degree of instability, primarily caused by GNSS satellite signal quality issues resulting from changes in the experimental environment. The GNSS positioning error remains within 1.0 meter for the vast majority of the time, but significant fluctuations occur in the epoch ranges near 285600-286200 and 287400-288000. These fluctuations are mainly caused by degraded satellite signal quality, worsened geometric distribution, or multipath effects. In contrast, the GNSS / INS fusion system successfully suppressed large fluctuations in GNSS over a specific time period by utilizing the short-term stability of the inertial navigation system (INS). Positioning errors were significantly reduced in all three directions, resulting in more stable and reliable overall performance. The advantages of GNSS / INS fusion are particularly evident when error fluctuations are large in the 287,400-288,000 epoch range, demonstrating good adaptability to signal interruptions and adverse conditions.

[0158] S3. Based on the fused navigation data, the two-dimensional visual observation data and the three-dimensional spatial point cloud data are fused to obtain the front view image depth map; and combined with the inverse calculation method of depth value, the national geodetic coordinates of the disease are determined using the front view image depth map.

[0159] As a preferred embodiment, the step of fusing two-dimensional visual observation data and three-dimensional spatial point cloud data based on fused navigation data to obtain a front view image depth map, and then using the front view image depth map to determine the national geodetic coordinates of the disease using a depth value inverse calculation method, includes the following steps:

[0160] S31. Perform spatiotemporal joint calibration on the camera that acquires two-dimensional visual observation data and the laser scanner that acquires three-dimensional spatial point cloud data, and solve for the camera extrinsic parameters.

[0161] S32. Based on the shooting angle and field of view of the two-dimensional visual observation data, the point cloud data corresponding to the two-dimensional visual observation data is selected from the 3D laser point cloud to obtain the selected point cloud data.

[0162] S33. By transforming the coordinates, the filtered point cloud data is converted to the pixel coordinate system of the two-dimensional visual observation data to obtain the depth map of the front view image;

[0163] In a preferred embodiment, the step of transforming the filtered point cloud data to the pixel coordinate system of the two-dimensional visual observation data through coordinate transformation to obtain the front view image depth map includes the following steps:

[0164] S331. Through geographic coordinate transformation, the coordinates of each filtered point cloud data in the preset world geodetic coordinate system are converted to the coordinates in the local horizontal coordinate system.

[0165] S332. Based on the attitude angle corresponding to the two-dimensional visual observation data at the shooting time, convert the coordinates of each filtered point cloud data in the local horizontal coordinate system to the coordinates in the IMU coordinate system.

[0166] S333. Transform the coordinates of each filtered point cloud data in the IMU coordinate system to the image space rectangular coordinate system, and calculate the coordinates of each filtered point cloud data in the image plane coordinate system using the collinearity equation.

[0167] S334. Convert the coordinates of each filtered point cloud data in the image plane coordinate system to the pixel coordinate system to achieve the fusion of the two-dimensional visual observation data and the filtered point cloud data, and obtain the front view image depth map.

[0168] S34. Based on the two-dimensional visual observation data, the pixel coordinates of the disease are determined by the YOLOv10 algorithm. The calculated depth value is determined by combining the fused navigation data and the depth map of the front view image. Based on the pixel coordinates of the disease and the outer side of the camera, the geodetic coordinates of the disease are calculated.

[0169] As a preferred embodiment, the steps of determining the disease pixel coordinates based on two-dimensional visual observation data using the YOLOv10 algorithm, determining the calculated depth value by combining fused navigation data and the front view image depth map, and calculating the country geodetic coordinates of the disease based on the disease pixel coordinates and the outer edge of the camera include the following steps:

[0170] S341. Using a deep neural network structure, road scene segmentation is performed on two-dimensional visual observation data to obtain the road surface part and the non-road surface part;

[0171] S342. Based on the road surface, the YOLOv10 algorithm is used to identify and locate targets in the region of interest on the road surface, and the detection results located in non-road surface areas are removed.

[0172] S343. Based on the pixel coordinates of the lesion and the fused navigation data, read the depth value at the corresponding position on the front view image depth map. Based on the pixel coordinates of the lesion and the outside of the camera, determine the geodetic coordinates of the lesion through inverse calculation of the depth value.

[0173] In a preferred embodiment, the step of reading depth values ​​at the corresponding positions on the front view image depth map based on the defect pixel coordinates and fused navigation data, and determining the geodetic coordinates of the defect by inverse calculation of the depth values ​​based on the defect pixel coordinates and the outer edge of the camera, includes the following steps:

[0174] S3431. Based on the coordinates of the defective pixels, read the depth value at the corresponding position in the depth map of the front view image;

[0175] S3432. Obtain the camera's intrinsic parameters and convert the pixel coordinates of the lesion into image plane coordinates;

[0176] S3433. Based on the image plane coordinates, and according to the depth value and the camera focal length, the three-dimensional coordinates of the disease in the image space rectangular coordinate system are derived using the principle of similar triangles.

[0177] S3434. Based on camera extrinsic parameters, convert the three-dimensional coordinates in the image space rectangular coordinate system into the disease coordinates in the IMU coordinate system;

[0178] S3435. Using the transformation matrix corresponding to the IMU attitude angle, convert the defect coordinates in the IMU coordinate system to the defect coordinates in the local horizontal coordinate system.

[0179] S3436. Convert the national geodetic coordinates of the two-dimensional visual observation data shooting point into Gaussian plane coordinates through Gaussian forward calculation, and superimpose the disease coordinates in the local horizontal coordinate system onto the Gaussian coordinates of the shooting point to obtain the Gaussian plane coordinates and elevation of the disease.

[0180] S3437. Convert the Gaussian plane coordinates to national geodetic coordinates through Gaussian inverse calculation to obtain the national geodetic coordinates of the disease.

[0181] It should be noted that to achieve high-precision location of defects, a depth image must first be generated. The depth image records the position of each pixel in the front view image and the distance from the target point corresponding to that pixel to the virtual camera center, i.e., the depth value. The depth value is obtained by transforming the point cloud data within a certain range of the area where the front view image is located into the image space Cartesian coordinate system of the front view image, and calculating the distance from the origin. Camera extrinsic parameters are obtained through spatiotemporal joint calibration of the scanner and camera. Based on these extrinsic parameters, the point cloud and image are fused to obtain the front view image depth map. This fusion process relies on the aforementioned GNSS / IMU positioning results to achieve association with geospatial data. The core of point cloud and image fusion is to accurately match the geometric spatial information of the 3D laser point cloud with the visual information of the 2D front view image, forming visual data with depth information. The 3D laser point cloud is three-dimensional spatial point cloud data, and the 2D front view image is two-dimensional visual observation data. The specific steps are as follows:

[0182] 1. Spatiotemporal joint calibration to obtain key parameters: First, spatiotemporal joint calibration is performed on the laser scanner and the camera. On the one hand, the synchronization relationship between the two in the time dimension is determined to ensure that the point cloud data and the image data correspond to the same inspection time. On the other hand, the camera extrinsic parameters, including the rotation matrix and translation vector, are solved to clarify the relative positional relationship between the camera coordinate system and the scanner coordinate system, laying the foundation for subsequent spatial coordinate association.

[0183] 2. Point cloud data preprocessing and region filtering: Based on the shooting angle and field of view of the front view image, the point cloud data corresponding to the front view image is filtered out from the massive 3D laser point cloud. Irrelevant area point clouds are removed to reduce the amount of computation, ensuring that the filtered point cloud and the image cover the same inspection scene, such as the road surface, guardrail, etc. of the same road section.

[0184] 3. Coordinate transformation to associate point cloud with image pixels: The following four steps are used to transform the point cloud from the global coordinate system to the image pixel coordinate system and establish a one-to-one correspondence between the point cloud and the image pixels: The transformation process is broken down into the following four steps.

[0185] 1) Converting 3D laser point cloud points to the local horizontal coordinate system in the WGS84 coordinate system. Let the object point, i.e., the laser point cloud point, have coordinates (X, Y, Z)WGS84 in the world geodetic coordinate system, and its position at the time the photo was taken be (X, Y, Z)O. The geodetic coordinates at the time the front view photo was taken are (B, L, H). The formula for calculating the coordinates [(X, Y, Z)Local] of the object point in the world geodetic coordinate system to the local horizontal coordinate system is as follows:

[0186] ;

[0187] ;

[0188] In the formula, R WGS84 This is the coordinate transformation matrix in the WGS-84 coordinate system.

[0189] 2) Transform the local horizontal coordinate system object point to the inertial navigation (IMU) coordinate system. After transforming to the local horizontal coordinate system, determine the attitude angle corresponding to the moment the front view image was captured. The coordinates (x, y, z) in the local horizontal coordinate system are converted to coordinates (x, y, z) in the IMU coordinate system. The calculation formula for the IMU is as follows:

[0190] ;

[0191] ;

[0192] In the formula, R Local This is the attitude transformation matrix in the local coordinate system.

[0193] 3) Transform from the IMU coordinate system to the image space rectangular coordinate system (Xp, Yp, Zp), and solve for the image plane coordinates (XC, YC) of the object point using the collinearity equation. The transformation from the IMU coordinate system to the image space rectangular coordinate system mainly involves 6 parameters: 3 rotation angles (α, β, γ) and 3 translations (ΔX, ΔY, ΔZ), and their calculation formulas are as follows:

[0194] ;

[0195] ;

[0196] ;

[0197] In the formula, R C The attitude transformation matrix, also known as the direction cosine matrix, is used in photogrammetry / visual navigation to describe the attitude transformation relationship between the image space coordinate system and the object space coordinate system. f is the principal distance of the camera (or photographic equipment).

[0198] 4) Transform from plane coordinate system to pixel coordinate system The image plane coordinate system has its origin at the camera's optical axis, while the pixel coordinate system has its origin at the top left corner of the image. The two are transformed using the pixel spacing p and the principal point coordinates (x0, y0), as shown in the formula:

[0199] ;

[0200] Each depth map is an image of the same size as the corresponding front view image. After the depth map and the front view image are superimposed, the row and column numbers of a point on the front view image are used to locate the corresponding position on the depth map, and the depth value at that position is obtained. Then, through coordinate transformation, the pixel values ​​are converted into 3D spatial point coordinates in the image space coordinate system. This process is the inverse operation of depth value calculation, and the principle is as follows: Figure 10 As shown, the depth map and image are as follows Figure 11 As shown.

[0201] 1) HRNet (High-Resolution Network) is used for road scene segmentation, dividing the forward-looking image into road and non-road portions. This effectively distinguishes road areas from non-road areas, providing accurate road information for subsequent defect identification. Secondly, the YOLOv10 algorithm is used to identify and locate regions of interest (ROIs) on the road surface. When a detected target is located outside the road, it is removed to avoid the impact of false detections of non-road targets on the identification results. Finally, small bounding box information is merged by combining scene constraints and target category information. Based on the road defects identified by AI, the pixel coordinates of the defects are obtained in the forward-looking image, and the RGB values ​​at the corresponding positions in the depth map are obtained and the depth value is calculated.

[0202] The road scene segmentation using HRNet (High-Resolution Network) involves: scaling and normalizing the road images acquired by the vehicle to a fixed resolution; employing the HRNet backbone network, leveraging its multi-resolution branch parallelism and cross-branch feature fusion structure to preserve high-resolution features and avoid the detail loss problem common in segmentation tasks; adding a multi-scale feature fusion module at the backbone network output, mapping to the number of road segmentation categories through 1×1 convolution, and then upsampling to the original image resolution to obtain a segmentation probability map; finally, thresholding and morphological operations (erosion / dilation) are used to optimize the segmentation results, outputting a road region mask.

[0203] Furthermore, the YOLOv10 algorithm is used to identify and locate Regions of Interest (ROIs) on the road surface. This includes scaling the road surface images acquired by the vehicle to a YOLOv10-compatible resolution (e.g., 640×640) and normalizing them; if LiDAR data needs to be fused, the point cloud can be projected as a depth map and RGB image stitched together as input. A lightweight YOLOv10 backbone network is adopted, retaining its anchor-free detector head and PAN-FPN feature fusion structure, and adjusting the detector head output to the corresponding category. CIoU loss is used to optimize bounding box regression, and FocalLoss is used to solve the class imbalance problem; data augmentation such as random flipping and illumination perturbation are combined, and the learning rate is adjusted using a cosine annealing strategy to complete the training. The model outputs the ROI category and bounding box coordinates, removes redundant boxes using NMS, and filters false detections by combining prior road surface knowledge, finally outputting accurate ROI identification and localization results.

[0204] 2) The final CGCS2000 coordinates are calculated using depth values, combined with the pixel coordinates of the lesions and the pose information of the photograph. This requires multiple steps of inverse coordinate transformation. The entire process is based on the spatial association information of the photograph (from GNSS / IMU). The specific process is as follows:

[0205] First, key parameters are prepared by acquiring camera intrinsic parameters (pixel spacing, principal point coordinates, obtained from camera calibration) and pose information at the time of image capture (including the geographic coordinates of the inspection equipment's CGCS2000 and IMU attitude angles, both from GNSS / IMU fusion positioning output). Simultaneously, the coordinates of defective pixels are obtained through AI identification, corresponding depth values ​​are read from the depth map, and the required coordinate transformation matrix is ​​pre-calculated based on the IMU attitude angles. Specifically, this includes:

[0206] Initiate the coordinate transformation process; based on camera intrinsic parameters, convert the defect pixel coordinates to image plane coordinates, correct the origin difference between the pixel coordinate system and the image plane coordinate system, and achieve the transition from pixel position to image plane position; combine depth value and camera focal length, and use the principle of similar triangles to derive the three-dimensional coordinates of the defect in the image space rectangular coordinate system, and clarify the spatial position of the defect relative to the virtual photography center; based on the camera extrinsic parameters obtained from spatiotemporal joint calibration, including rotation matrix and translation vector, convert the image space coordinates to IMU coordinate system coordinates, so that the defect coordinates are bound to the motion attitude of the inspection equipment; use the transformation matrix corresponding to the IMU attitude angle. The disease coordinates in the IMU coordinate system can be converted to local horizontal coordinate system coordinates through inverse matrix operations. This coordinate system has the photo shooting point as the origin and north, east, and sky as coordinate axes. The CGCS2000 coordinates of the photo shooting point are converted to Gaussian plane coordinates through Gaussian forward calculation. Then, the disease coordinates in the local horizontal coordinate system are superimposed on the Gaussian coordinates of the shooting point to obtain the Gaussian plane coordinates and elevation of the disease. Finally, the Gaussian plane coordinates are converted to CGCS2000 geodetic coordinates through Gaussian inverse calculation to complete the final CGCS2000 coordinate calculation of the disease. The CGCS2000 coordinates are the national geodetic coordinates.

[0207] Specifically, the absolute coordinates of the inspection system are obtained through the integrated navigation system, and the coordinates are transmitted to the target object based on the point cloud and image calibration parameters, thereby achieving high-precision positioning of road surface defects with a positioning error of centimeters.

[0208] like Figure 12-13 Scene 1, shown, depicts the front of a building. This scene is well-lit with few shadows and rich facade features. The building's front is a regular rectangular structure with distinct geometric characteristics. Numerous target points are evenly distributed across the building's surface and the ground. While the building's geometric features are clear, the checkerboard-like target points are evenly distributed, and the building boundaries almost completely overlap, resulting in good blending. However, the features on both sides are relatively simple, lacking features at long distances, making it impossible to test the blending effect at a distance.

[0209] Ten checkerboard markers were extracted from the image data as checkpoints, and ten corresponding checkpoints were selected from the point cloud data. The pixel coordinates of each checkpoint on the corresponding image were calculated. As shown in Table 1, the overall error of the fusion result between the point cloud and the image for Scene 1 is 2-4 pixels, indicating high overall fusion accuracy, but it cannot be verified for distant areas.

[0210] Table 1. Results of point cloud and image fusion in Scene 1

[0211] The building's front view achieved high fusion accuracy using evenly distributed calibration points, with reprojection errors controlled within the range of 2–4 pixels, indicating good matching between the image and point cloud at close range. Although the overall correction was accurate, the distant features were relatively sparse, failing to fully reflect the matching performance at long distances. Future research could consider adding distant calibration points to improve the overall evaluation.

[0212] like Figure 14-15 Scene 2, as shown, comprises a colonnaded building with a facade characterized by regular columns and a top beam. Calibration points are widely distributed on the ground, with walls, a facade, and some ancillary facilities on the left and right sides, respectively. The columns and walls provide good geometric feature references. The scene exhibits significant depth variations, making it suitable for testing the algorithm's accuracy in the depth direction. The drawback is that the columns and beams may cause occlusion, increasing the complexity of depth fusion.

[0213] As can be seen from Table 2, the overall error of the fusion result of point cloud and image in scene 2 is between 3 and 6 pixels, indicating that the depth fusion still maintains medium accuracy over a large depth range.

[0214] Table 2. Results of point cloud and image fusion in Scene 2

[0215] The corridor scene utilizes prominent pillars and beams to form a regular geometric structure, keeping the reprojection error between the image and the point cloud within 3–6 pixels over a significant depth range. This result reflects that in environments with drastic depth variations, although correction is relatively stable in most areas, local occlusion and edge effects can still interfere with accuracy, requiring further improvement of the data fusion strategy.

[0216] like Figure 16-17 Scene 3, as shown, is primarily an open road with buildings located in the distance, featuring railings on both sides of the facade. Calibration points and cubes are arranged on the ground, providing reference for height and distance measurements. The scene offers a wide field of view and strong sense of depth. This expansive scene is suitable for testing the fusion effect of depth maps and images in large-scale scenes. The widely distributed calibration points allow for testing depth fusion accuracy at different distances. However, lighting conditions may be significantly affected by environmental factors, as shown in Table 3.

[0217] Table 3. Results of point cloud and image fusion in Scene 3

[0218] The open road scene covers multiple layers of near, mid, and far distances, and its fusion results show an error between 2 and 5 pixels, reflecting the difference in matching performance at different distances. Overall correction is relatively ideal, but the error is slightly larger in weak feature areas such as distant buildings and railings, suggesting that further optimization of long-distance feature extraction and matching methods should be strengthened to improve the robustness of global fusion. In general, the three scenes each have their own characteristics: the building facade scene is suitable for verifying local high-precision fusion, but not for long-distance evaluation; the building corridor scene can better reflect the depth fusion capability, but there are local accuracy fluctuations due to occlusion; the open road scene covers a wide depth range, testing the fusion accuracy at both near and far distances, but the overall error is relatively high.

[0219] This data provides a basis for the subsequent selection and optimization of fusion algorithms, allowing for a balance between local and global fusion accuracy requirements based on specific application scenarios.

[0220] 1) Based on the AI ​​recognition results, obtain the pixel coordinates of the lesions in the front view image, obtain the RGB values ​​at the corresponding positions in the depth map, and calculate the depth value in reverse.

[0221] 2) The final CGCS2000 coordinates are calculated by combining the depth value with the pixel coordinates of the lesion and the pose information of the photo.

[0222] The absolute coordinates of the inspection system are obtained by combining the navigation system. Based on the point cloud and image calibration parameters, the coordinates are transmitted to the target object, thereby achieving high-precision positioning of road defects with a positioning error of centimeters.

[0223] S4. Obtain the starting kilometer marker number during the highway inspection process, and combine it with the national geodetic coordinates of the defects to determine the mileage marker number of the defects using the Gaussian forward algorithm.

[0224] As a preferred embodiment, the step of obtaining the starting kilometer marker number during highway inspection and determining the mileage marker number of the road damage using the Gaussian forward algorithm, in conjunction with the national geodetic coordinates of the road damage, includes the following steps:

[0225] S41. Obtain the starting kilometer marker number and the national geodetic coordinates of the starting kilometer marker number during the highway inspection process.

[0226] S42. Construct an initial base map containing the latitude and longitude coordinates of the road network station numbers and lane lines. Combine this with the national geodetic coordinates of the starting kilometer station number. On the initial base map, draw perpendicular lines from the starting kilometer station number and the defect to the center line of the highway lane, respectively, to obtain the first intersection point and the second intersection point.

[0227] S43. Calculate the national geodetic coordinates of the first and second intersection points in the national geodetic coordinate system using the Gaussian forward calculation formula;

[0228] S44. Based on the idea of ​​differentiation, and according to the national geodetic coordinates of the first and second intersection points, calculate the curve distance between the first and second intersection points, and determine the mileage station number of the defect according to the starting kilometer station number.

[0229] As a preferred embodiment, the step of calculating the curve distance between the first and second intersection points based on the idea of ​​differentiation and according to the national geodetic coordinates of the first and second intersection points in the national geodetic coordinate system, and determining the mileage station of the defect according to the starting kilometer marker number, includes the following steps:

[0230] S441. Divide the curve between the first intersection point and the second intersection point into several sub-intervals;

[0231] S442. For each subinterval, use the geodetic coordinates of its endpoints or representative points within the interval to calculate the approximate local arc length of that subinterval.

[0232] S443. Sum the approximate local arc length values ​​of all sub-intervals to obtain the curve distance between the first intersection point and the second intersection point, and determine the mileage station number corresponding to the location of the defect based on the starting kilometer station number.

[0233] It should be noted that mileage markers play a crucial role as road landmarks in highway inspection operations. Traditional inspection methods often rely on mileage markers to define inspection areas and tasks, but this is easily limited by human error, equipment limitations, and complex terrain. However, in actual road conditions, mileage markers are not always precisely distributed at standard 100-meter intervals, nor do they correspond one-to-one with each kilometer marker.

[0234] Specifically, when determining the mileage marker of the defect, assuming the starting mileage marker is K100 with geographical coordinates (B1, L1, H1), and given that the geographical coordinates of any point B are (B2, L2, H2), the specific technical solution for calculating the mileage marker at point B is as follows:

[0235] 1) Draw perpendicular lines from the starting kilometer marker and target point B to the road centerline, intersecting the road centerline at points A and C. The geographical coordinates of points A and C are (B3, L3, H3) and (B4, L4, H4), respectively. Figure 18 As shown;

[0236] 2) Calculate the coordinates of points A and C in CGCS2000 using the Gaussian forward calculation formula. The calculation formula is shown below:

[0237] ;

[0238] ;

[0239] ;

[0240] ;

[0241] ;

[0242] ;

[0243] ;

[0244] ;

[0245] ;

[0246] In the formula, B is the latitude of the point. Let be the second eccentricity of the ellipse, a be the major semi-axis of the ellipsoid, and b be the minor semi-axis of the ellipsoid. L is the longitude of the point, L0 is the longitude of the central meridian, (x,y) are the Gaussian plane rectangular coordinates in the CGCS2000 coordinate system, x is the vertical axis (parallel to the central meridian) coordinate, y is the horizontal axis (perpendicular to the central meridian) coordinate, N is the radius of curvature of the ellipsoid in the ramustralis direction at a certain latitude, reflecting the lateral curvature of the ellipsoid at that point, ρ′′ is the conversion coefficient between radians and seconds, used to convert the radian value of an angle to a second value, and is a commonly used constant for angle unit conversion in geodesy, l′′ is the second value of the longitude difference, that is, the difference between the longitude L of the point and the longitude L0 of the central meridian (l=L-L0) converted to a second value, t is the tangent of latitude, i.e., t=tanB, which is an intermediate variable used to simplify calculations in the Gaussian forward calculation formula, η is an auxiliary quantity related to the second eccentricity, e is the first eccentricity of the ellipsoid, X is the meridian arc length, and the formula for calculating the meridian arc length is as follows:

[0247] ;

[0248] Where a0, a2, a3, a4, a6, and a8 are fundamental constants, i.e., coefficients related to the ellipsoidal parameters, calculated according to the following formula:

[0249] ;

[0250] In the formula, m0, m2, m3, m4, m6, and m8 are basic constants, calculated according to the following formula:

[0251] ;

[0252] In the formula, a is the semi-major axis of the Earth's ellipsoid.

[0253] Therefore, the coordinates of points A and C in CGCS2000 can be obtained as A( ), C ( ).

[0254] 3) Solving for the length of curve AC: Based on the idea of ​​differentiation, assume that n-1 numbers are used to divide the interval [ , The interval is divided into n subintervals, and the arc length of each subinterval is approximately expressed by the following formula:

[0255] ;

[0256] In the formula, This represents the approximate local arc length of the i-th subinterval. This represents the coordinate increment along the x-axis, specifically the coordinate increment of the i-th subinterval along the x-axis. This represents the coordinate increment along the y-axis, specifically the coordinate increment of the i-th subinterval along the y-axis. Indicates the curve at x i The derivative at point X, the first derivative of the curve function y=f(x), This represents the x-coordinate of a representative point, that is, the x-coordinate of a representative point within the i-th subinterval.

[0257] The total arc length of the curve is approximately equal to the sum of the arc lengths of its individual subintervals, and its calculation formula is as follows:

[0258] ;

[0259] The distance between points A and C can be calculated as S. The mileage marker at point C can be determined as K100+S. From this, the mileage marker at the target point B can be determined, and it can be judged whether the kilometer marker has been reached.

[0260] In addition, to improve the accuracy and intelligence of inspection operations and reduce human error, automatic pile calibration technology has emerged. Combining target geolocation and mileage marker matching technology, automatic pile calibration technology can automatically identify and correct the pile position based on the location of the inspection equipment, ensuring consistency between the pile number and the actual location, and optimizing inspection efficiency and management accuracy.

[0261] Automatic mileage marker calibration technology refers to the automatic matching and correction of inspection equipment with target mileage markers by using GNSS / INS fusion high-precision positioning technology on vehicle-mounted hardware and automatically interfacing with the mileage marker data. This technology can solve problems such as positional deviations and error accumulation associated with traditional manual recording of mileage markers, and improve the accuracy of inspection work. Vehicle-mounted data acquisition and control equipment plays a crucial role in automatic mileage marker calibration technology. By collecting high-precision positioning information and driving status data of the inspection equipment in real time, it dynamically monitors the vehicle's trajectory and matches it with the target mileage marker. Simultaneously, the vehicle-mounted equipment can also record road anomaly information, providing support for subsequent data analysis and correction.

[0262] Based on the latitude and longitude of the target point, the road centerline, and the location of kilometer markers, the kilometer marker at any location can be accurately calculated. The specific steps are as follows:

[0263] Step 1: Construct the initial base map of the road network's stationing coordinates and lane lines;

[0264] Step 2: Calculate the vehicle's mileage and output real-time latitude and longitude based on the GNSS / INS integrated navigation system;

[0265] Step 3: Compare the current vehicle location with the preset kilometer marker geographical coordinates to determine whether the vehicle has reached the next kilometer marker location; if not, return to Step 2 to continue updating the location information in real time.

[0266] Step 4: Determine whether the deviation between the vehicle's mileage and the actual kilometer marker exceeds a threshold. When the vehicle reaches a kilometer marker, compare the difference between the mileage calculated by the integrated navigation system and the actual kilometer marker mileage. If the deviation exceeds a preset threshold, it is determined that there is a kilometer marker misalignment or cumulative error.

[0267] Step 5: Execute the automatic kilometer marker calibration program. Start the automatic kilometer marker calibration algorithm, and correct the cumulative mileage error based on the current precise location of the vehicle and the actual kilometer marker geographical coordinates. This achieves dynamic matching and synchronization between the vehicle's mileage and the actual kilometer marker, thus completing the kilometer marker calibration operation.

[0268] In the post-processing stage, the station data on the base map played a crucial role in reference and correction. By comparing the inspection data with the station positions on the base map, the station matching accuracy was further optimized, and the accumulation of deviations was reduced. The base map station numbers also provided a unified coordinate benchmark for the accumulation of long-term series data, facilitating subsequent analysis and trend research.

[0269] According to a second embodiment of the present invention, a highway inspection target positioning system based on multi-source data fusion is provided, the system comprising the following steps:

[0270] The data acquisition module is used to acquire multi-source data, including satellite positioning data, inertial sensor data, two-dimensional visual observation data and three-dimensional spatial point cloud data, based on preset multi-source sensors during highway inspection.

[0271] The data fusion module is used to determine the fused navigation data based on the acquired satellite positioning data and inertial sensor data, through deep coupling of satellite positioning carrier double-difference observation and inertial navigation calculation of inertial sensors, and combined with Kalman filter recursive error correction.

[0272] The disease identification module is used to fuse two-dimensional visual observation data and three-dimensional spatial point cloud data based on the fused navigation data to obtain a front view image depth map; and combined with the inverse calculation method of depth value, the front view image depth map is used to determine the national geodetic coordinates of the disease.

[0273] The defect location module is used to obtain the starting kilometer marker number during highway inspection and, in conjunction with the national geodetic coordinates of the defect, uses the Gaussian forward algorithm to determine the mileage marker number of the defect.

[0274] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for locating highway inspection targets based on multi-source data fusion, characterized in that, The method includes the following steps: S1. Based on preset multi-source sensors, acquire multi-source data including satellite positioning data, inertial sensor data, two-dimensional visual observation data and three-dimensional spatial point cloud data during highway inspection. S2. Based on the acquired satellite positioning data and inertial sensor data, the fused navigation data is determined by deeply coupling the satellite positioning carrier double-difference observation with the inertial navigation solution of the inertial sensor, and by combining the Kalman filter recursive error correction. S3. Based on the fused navigation data, the two-dimensional visual observation data and the three-dimensional spatial point cloud data are fused to obtain the front view image depth map; and combined with the inverse calculation method of depth value, the national geodetic coordinates of the disease are determined using the front view image depth map. S4. Obtain the starting kilometer marker number during the highway inspection process, and combine it with the national geodetic coordinates of the defects to determine the mileage marker number of the defects using the Gaussian forward algorithm.

2. The highway inspection target localization method based on multi-source data fusion according to claim 1, characterized in that, The process of determining the fused navigation data based on the acquired satellite positioning data and inertial sensor data, through deep coupling of satellite positioning carrier double-difference observation and inertial navigation calculation by inertial sensors, and combined with Kalman filter recursive error correction, includes the following steps: S21. Based on the acquired satellite positioning data, the pseudorange double difference value and the carrier phase double difference value are calculated respectively using the satellite positioning carrier observation model and the carrier double difference equation. S22. Based on inertial sensor data and using the INS mechanical arrangement equations to perform inertial navigation calculations, the current state vector containing position, velocity and attitude estimates is obtained at the current moment. Combined with the state transition matrix, the predicted state vector and covariance matrix for the next moment are predicted. S23. Based on the current position, velocity, and attitude estimates, and combined with the satellite ephemeris and base station location, calculate the double-difference geometric distance predicted by the INS. S24. Subtract the pseudorange double difference value and the carrier phase double difference value from the double difference geometric distance respectively to form the double difference observation residual. Combine the residual with the pre-configured measurement equation to calculate the Kalman gain and form the gain matrix. S25. Using the gain matrix, the predicted state vector is corrected to obtain the optimal estimates of the inertial component errors and navigation parameter errors; the inertial component errors include gyroscope drift and accelerometer bias; the navigation parameter errors include attitude angle errors, velocity errors, and position errors. S26. Using the optimal estimates of inertial component errors and navigation parameter errors, perform dual feedback correction on the inertial sensor data to obtain fused navigation data.

3. The method for locating highway inspection targets based on multi-source data fusion according to claim 2, characterized in that, The process of using the optimal estimates of inertial component errors and navigation parameter errors to perform dual feedback correction on inertial sensor data to obtain fused navigation data includes the following steps: S251. Convert the attitude error into an attitude correction value through coordinate transformation, and update the attitude matrix of the INS to obtain the updated attitude data. S252. The velocity error and position error are superimposed on the velocity and position calculation results of the INS to eliminate the cumulative deviation of navigation parameters and obtain updated velocity and position data. S253. Integrate the updated attitude data with the updated velocity and position data to obtain fused navigation data.

4. The highway inspection target localization method based on multi-source data fusion according to claim 1, characterized in that, The process of fusing two-dimensional visual observation data and three-dimensional spatial point cloud data based on fused navigation data to obtain a front view image depth map, and then using the depth map to determine the national geodetic coordinates of the disease using a depth value inverse calculation method, includes the following steps: S31. Perform spatiotemporal joint calibration on the camera that acquires two-dimensional visual observation data and the laser scanner that acquires three-dimensional spatial point cloud data, and solve for the camera extrinsic parameters. S32. Based on the shooting angle and field of view of the two-dimensional visual observation data, the point cloud data corresponding to the two-dimensional visual observation data is selected from the 3D laser point cloud to obtain the selected point cloud data. S33. By transforming the coordinates, the filtered point cloud data is converted to the pixel coordinate system of the two-dimensional visual observation data to obtain the depth map of the front view image; S34. Based on the two-dimensional visual observation data, the pixel coordinates of the disease are determined by the YOLOv10 algorithm. The calculated depth value is determined by combining the fused navigation data and the depth map of the front view image. Based on the pixel coordinates of the disease and the outer side of the camera, the geodetic coordinates of the disease are calculated.

5. The method for locating highway inspection targets based on multi-source data fusion according to claim 4, characterized in that, The process of transforming the filtered point cloud data to the pixel coordinate system of two-dimensional visual observation data to obtain the front view image depth map includes the following steps: S331. Through geographic coordinate transformation, the coordinates of each filtered point cloud data in the preset world geodetic coordinate system are converted to the coordinates in the local horizontal coordinate system. S332. Based on the attitude angle corresponding to the two-dimensional visual observation data at the shooting time, convert the coordinates of each filtered point cloud data in the local horizontal coordinate system to the coordinates in the IMU coordinate system. S333. Transform the coordinates of each filtered point cloud data in the IMU coordinate system to the image space rectangular coordinate system, and calculate the coordinates of each filtered point cloud data in the image plane coordinate system using the collinearity equation. S334. Convert the coordinates of each filtered point cloud data in the image plane coordinate system to the pixel coordinate system to achieve the fusion of the two-dimensional visual observation data and the filtered point cloud data, and obtain the front view image depth map.

6. The highway inspection target localization method based on multi-source data fusion according to claim 4, characterized in that, The process of determining the pixel coordinates of the lesion based on two-dimensional visual observation data using the YOLOv10 algorithm, determining the calculated depth value by combining fused navigation data and the depth map of the foreground image, and calculating the geodetic coordinates of the lesion based on the pixel coordinates of the lesion and the outer edge of the camera includes the following steps: S341. Using a deep neural network structure, road scene segmentation is performed on two-dimensional visual observation data to obtain the road surface part and the non-road surface part; S342. Based on the road surface, the YOLOv10 algorithm is used to identify and locate targets in the region of interest on the road surface, and the detection results located in non-road surface areas are removed. Combined with the preset scene constraints and the target category information, the small box information is merged, and the pixel coordinates of the defects in the two-dimensional visual observation data are output. S343. Based on the pixel coordinates of the lesion and the fused navigation data, read the depth value at the corresponding position on the front view image depth map. Based on the pixel coordinates of the lesion and the outside of the camera, determine the geodetic coordinates of the lesion through inverse calculation of the depth value.

7. The highway inspection target localization method based on multi-source data fusion according to claim 6, characterized in that, The process of reading depth values ​​at corresponding positions on the front view image depth map based on the defect pixel coordinates and fused navigation data, and determining the geodetic coordinates of the defect through inverse calculation of the depth values ​​based on the defect pixel coordinates and the outer edge of the camera, includes the following steps: S3431. Based on the coordinates of the defective pixels, read the depth value at the corresponding position in the depth map of the front view image; S3432. Obtain the camera's intrinsic parameters and convert the pixel coordinates of the lesion into image plane coordinates; S3433. Based on the image plane coordinates, and according to the depth value and the camera focal length, the three-dimensional coordinates of the disease in the image space rectangular coordinate system are derived using the principle of similar triangles. S3434. Based on camera extrinsic parameters, convert the three-dimensional coordinates in the image space rectangular coordinate system into the disease coordinates in the IMU coordinate system; S3435. Using the transformation matrix corresponding to the IMU attitude angle, convert the defect coordinates in the IMU coordinate system to the defect coordinates in the local horizontal coordinate system. S3436. Convert the national geodetic coordinates of the two-dimensional visual observation data shooting point into Gaussian plane coordinates through Gaussian forward calculation, and superimpose the disease coordinates in the local horizontal coordinate system onto the Gaussian coordinates of the shooting point to obtain the Gaussian plane coordinates and elevation of the disease. S3437. Convert the Gaussian plane coordinates to national geodetic coordinates through Gaussian inverse calculation to obtain the national geodetic coordinates of the disease.

8. The method for locating highway inspection targets based on multi-source data fusion according to claim 1, characterized in that, The process of obtaining the starting kilometer marker number during highway inspection and determining the mileage marker number of the road damage using the Gaussian forward algorithm, in conjunction with the national geodetic coordinates of the road damage, includes the following steps: S41. Obtain the starting kilometer marker number and the national geodetic coordinates of the starting kilometer marker number during the highway inspection process. S42. Construct an initial base map containing the latitude and longitude coordinates of the road network station numbers and lane lines. Combine this with the national geodetic coordinates of the starting kilometer station number. On the initial base map, draw perpendicular lines from the starting kilometer station number and the defect to the center line of the highway lane, respectively, to obtain the first intersection point and the second intersection point. S43. Calculate the national geodetic coordinates of the first and second intersection points in the national geodetic coordinate system using the Gaussian forward calculation formula; S44. Based on the idea of ​​differentiation, and according to the national geodetic coordinates of the first and second intersection points, calculate the curve distance between the first and second intersection points, and determine the mileage station number of the defect according to the starting kilometer station number.

9. A method for locating highway inspection targets based on multi-source data fusion according to claim 8, characterized in that, The method based on the idea of ​​differentiation, and calculating the curve distance between the first and second intersection points according to the national geodetic coordinates of the first and second intersection points, and determining the mileage of the defect according to the starting kilometer marker number, includes the following steps: S441. Divide the curve between the first intersection point and the second intersection point into several sub-intervals; S442. For each subinterval, use the geodetic coordinates of its endpoints or representative points within the interval to calculate the approximate local arc length of that subinterval. S443. Sum the approximate local arc length values ​​of all sub-intervals to obtain the curve distance between the first intersection point and the second intersection point, and determine the mileage station number corresponding to the location of the defect based on the starting kilometer station number.

10. A method for locating highway inspection targets based on multi-source data fusion according to claim 9, characterized in that, The formula for calculating the approximate local arc length of each sub-interval using the geodetic coordinates of its endpoints or representative points within the interval is as follows: ； In the formula, This represents the approximate local arc length of the i-th subinterval. This represents the coordinate increment along the x-axis. This represents the coordinate increment along the y-axis. Indicates the curve at x i The derivative at point, This represents the x-coordinate of the point.

Images