An air-ground integrated traffic facility intelligent patrol method and system based on a large model

Abstract

This invention discloses an intelligent inspection method and system for integrated air-ground transportation facilities based on a large model. The method utilizes the collaborative operation of ground inspection vehicles and aerial drones to simultaneously inspect and collect data from the road section to be inspected, acquiring both vehicle-mounted and airborne data. Based on an artificial intelligence model, edge inference is performed to output candidate abnormal events. Through time window alignment and coarse positioning filtering, vehicle-mounted and airborne data are matched, and bidirectional identification verification is performed to confirm the credibility of abnormal events. Further calibration and measurement of abnormal events are conducted to compensate for positioning deviations. Duplicate abnormal events are deduplicated through coarse registration and fine matching clustering, outputting accurate positioning results. The system includes modules such as an air-ground collaborative data acquisition unit, a spatiotemporal alignment unit, an edge inference unit, and a mutual verification judgment unit. It can improve the efficiency and accuracy of transportation facility inspections, while supporting closed-loop management and verification of abnormal events, thus enhancing the control capabilities of transportation facilities and road surface defects.

Description

Technical Field

[0001] This invention relates to the field of road inspection technology, and more specifically to an intelligent inspection method and system for air-ground integrated transportation facilities based on a large model. Background Technology

[0002] Inspections of road traffic facilities and pavement defects are fundamental to highway maintenance and operation management. Inspection results are typically used in hazard identification, repair dispatching, maintenance assessments, and road condition statistics. Traditional inspection methods primarily rely on manual on-site inspections or vehicle-mounted checks, depending on the observer's observation and experience. This approach suffers from high labor intensity, low coverage efficiency, subjective results, and difficulty in ensuring granular and consistent recording. With the expansion of road networks, increased inspection frequency, and growing demand for refined maintenance, the industry is gradually introducing vehicle-mounted camera data collection, drone aerial photography, and AI-based image recognition technologies to improve inspection efficiency and reduce labor costs.

[0003] In existing technologies, one approach involves using ground inspection vehicles equipped with cameras to acquire road surface videos or images, and then employing deep learning models to automatically identify and issue warnings for targets such as cracks, potholes, repair marks, damaged road markings, abnormal guardrails, and spilled materials. Another approach uses drones to conduct aerial inspections of roads or key infrastructure areas, thereby increasing inspection coverage and obtaining overall situational information from an overhead perspective. While these technical approaches improve inspection efficiency to some extent, they still have significant technical shortcomings during practical implementation, especially in "air-ground collaborative" scenarios.

[0004] First, there are significant differences in perspective and scale between aerial and ground-based data collection: drones typically have higher altitudes and a superior viewpoint, while ground inspection vehicles are limited by their altitude and road obstructions. This results in differences in imaging scale, perspective distortion, and texture detail for the same anomalous target. Recognition models trained or inferred based on a single perspective are prone to inconsistencies in class classification, significant differences in confidence levels, and missed / false detections under cross-perspective conditions. Due to the lack of consistency constraints and cross-validation mechanisms for recognition results at both the air and ground levels, existing systems often rely on manual verification to confirm anomalous events, thus weakening the actual effectiveness of automated patrols.

[0005] Secondly, the "quantifiability" and "dispatchability" of inspection results remain insufficient. Road defects and facility anomalies often require not only category identification but also the output of quantitative parameters such as length, width, area, and damage ratio, as well as accurate location, to support maintenance priority assessment, material and labor time estimation, maintenance acceptance, and subsequent statistical evaluation. However, under actual data collection conditions, the stability of quantitative parameters in single-end identification results is poor due to factors such as shooting height, posture changes, lighting, and road surface material. Deviations exist in the size estimation of the same target output from different terminals and perspectives, making it difficult to form traceable and consistent measurement results. Consequently, the problem of "identifying but being unable to directly dispatch a work order" remains widespread.

[0006] Furthermore, positioning errors and duplicate reporting issues hinder the quality management of patrol data. Existing patrol systems typically rely on GPS coordinates, mileage markers, or simple geographic labeling for anomaly location. However, in road environments, factors such as GPS signal errors, obstruction, and multipath effects can cause positioning drift. When the same anomaly is collected and reported multiple times at different times and from different terminals, using only coarse-grained correlation based on coordinates / mileage markers can easily split the same anomaly into multiple events, leading to duplicate dispatches, duplicate verifications, and statistical distortions. Conversely, if the positioning drift is significant, it may result in inaccurate dispatch locations, increasing on-site search and verification costs, and even affecting the timeliness of response.

[0007] Therefore, for the application scenario of collaborative patrol by drones and inspection vehicles, there is an urgent need for a technical solution that can ensure the consistency of identification results at both the air and ground ends, reliable quantitative calibration, and deduplication positioning output under the conditions of differences in perspective / scale and positioning errors. This would improve the credibility, measurability, and dispatchability of patrol results, thereby supporting the closed-loop processing and refined management needs of road facility patrols. Summary of the Invention

[0008] To address the aforementioned issues, this invention provides a method and system for intelligent inspection of air-ground integrated transportation facilities based on a large model.

[0009] To achieve the above objectives, the present invention provides the following technical solution, which mainly includes:

[0010] A method for intelligent inspection of air-ground integrated transportation facilities based on a large model includes:

[0011] S1. Obtain patrol tasks and determine the road sections to be patrolled;

[0012] S2. Simultaneously patrol and collect data on the road section to be inspected using ground inspection vehicles and aerial drones, obtain vehicle-mounted data and airborne data, and bind time information and location-related information to the vehicle-mounted data and airborne data respectively.

[0013] S3. On the ground inspection vehicle and the airborne drone, respectively, edge reasoning is performed on the vehicle-mounted data and airborne data based on the artificial intelligence recognition model to output a candidate abnormal event set. The candidate abnormal event set includes at least the category information of traffic facility abnormalities or road surface defects and the corresponding target area information.

[0014] S4. Based on the time information and the location-related information, the candidate abnormal events output by the ground inspection vehicle terminal and the candidate abnormal events output by the aerial drone terminal are aligned and matched, and bidirectional identification and verification are performed to determine the abnormal events that have passed mutual verification.

[0015] S5. Perform calibration measurement based on the abnormal event of the mutual verification, and at least correct the quantitative measurement parameters and positioning parameters of the abnormal event;

[0016] S6. Perform coarse registration on abnormal events based on the positioning error range, and further perform fine matching based on the characteristics of the abnormal events themselves and the characteristics of the environment in which the abnormal events are located, so as to deduplicatize and aggregate duplicate abnormal events, and output the deduplicated abnormal events and their final positioning results.

[0017] S7. Generate inspection results and output them to the backend management platform.

[0018] In a specific embodiment, when S1 coordinates the planning of the inspection path of the ground inspection vehicle and the inspection path of the aerial drone, it includes at least: dividing the inspection task into multiple task segments according to the mileage range of the road segment to be inspected; assigning the driving path of the ground inspection vehicle and the flight path of the aerial drone to each task segment; and setting a lateral blind spot filling route and a circling verification route around the suspected abnormal location for the aerial drone related to the driving direction of the ground inspection vehicle, so as to obtain airborne acquisition data from different perspectives.

[0019] In a specific embodiment, the time and location information bound to the vehicle-mounted and airborne data in S2 includes at least: timestamp, frame number; and at least one of global positioning coordinates, flight altitude, heading angle, pitch angle, and roll angle; and the ground inspection vehicle and the airborne UAV end use a unified time synchronization mechanism to ensure that the timestamps of both ends fall on the same time base for alignment and matching in S4.

[0020] In a specific embodiment, each candidate abnormal event output by S3 includes at least: abnormal category, confidence level, target area information, collection timestamp, coarse location information, and event feature vector;

[0021] The event feature vector is obtained by the output of the intermediate feature layer corresponding to the target region by the artificial intelligence recognition model and by performing feature extraction on the target region; the feature extraction includes at least morphological features and / or texture features.

[0022] In a specific embodiment, the alignment matching and bidirectional recognition verification of S4 includes at least the following: within a preset time window Δt, screening candidate pairs of vehicle-mounted candidate abnormal events and airborne candidate abnormal events; for each candidate pair, if the abnormality categories are consistent and the coarse positioning distance between the two is less than a preset distance threshold Δd, then calculating the similarity of the event feature vectors; when the similarity is greater than a preset threshold θ, the candidate pair is determined as an abnormal event that has passed mutual verification; otherwise, the candidate pair is determined as an abnormal event to be reviewed and a review strategy is triggered; wherein, the review strategy includes at least one of the following: instructing an aerial UAV to perform supplementary image acquisition at the location of the abnormal event to be reviewed and / or instructing a ground inspection vehicle to reduce its speed and increase its sampling density at the location of the abnormal event to be reviewed.

[0023] In a specific embodiment, the calibration measurement of S5 includes at least one of quantitative measurement calibration and positioning calibration: Quantitative measurement calibration includes: for mutually verified abnormal events, calculating pixel-to-actual-scale conversion coefficients on both the vehicle-mounted and airborne ends based on camera intrinsic parameters, attitude information, and shooting height information, and performing consistency correction on the conversion coefficients based on the mutually verified abnormal events, thereby obtaining at least one of the corrected quantitative measurement parameters of crack width, crack length, pit area, and defect ratio; Positioning calibration includes: for mutually verified abnormal events, calculating the deviation vector between computer-based coarse positioning and vehicle-mounted coarse positioning, and performing deviation compensation on the coarse positioning information of subsequent abnormal events to reduce final positioning drift.

[0024] A large-scale model-based intelligent inspection system for integrated air-ground transportation facilities includes:

[0025] The air-ground collaborative data acquisition unit is used for simultaneous inspection and data collection of the road section to be inspected by ground inspection vehicles and airborne drones, to obtain vehicle-mounted data and airborne data, and to bind time information and location-related information to the vehicle-mounted data and airborne data.

[0026] The spatiotemporal alignment unit is used to align the time windows of data collected at both ends of the air and ground based on a unified time reference, and to perform candidate pairing and filtering based on coarse positioning information;

[0027] The edge inference unit is used to perform edge inference on the collected data based on the artificial intelligence recognition model at both the ground inspection vehicle end and the air drone end, and output a set of candidate abnormal events; each candidate abnormal event includes at least the abnormality category, confidence level, target area information, timestamp, coarse localization information and event feature vector.

[0028] The mutual verification determination unit is used to perform category consistency determination and event feature similarity calculation on candidate pairings, and output mutual verification pass abnormal events and pending review abnormal events based on preset thresholds; the calibration measurement unit is used to correct the quantitative measurement parameters and / or positioning parameters using mutual verification pass abnormal events as calibration samples.

[0029] The deduplication localization unit is used to perform coarse registration based on the localization error range, and further perform fine matching based on the characteristics of the abnormal event itself and / or the characteristics of the environment where the abnormal event is located, to deduplicate and aggregate duplicate abnormal events and output the final localization result;

[0030] The results output unit is used to generate inspection results and output them to the backend management platform.

[0031] In a specific embodiment, the spatiotemporal alignment unit includes a clock synchronization module and a time window index module; the clock synchronization module is used to ensure that the timestamps of the ground inspection vehicle and the airborne UAV meet a unified time reference; the time window index module is used to establish a correspondence between the vehicle-mounted data and the airborne data with a preset time window Δt, so as to support the candidate pairing screening and mutual verification judgment.

[0032] In a specific embodiment, the deduplication localization unit includes a coarse registration module and a fine matching clustering module: the coarse registration module is used to establish a spatial neighborhood candidate set for abnormal events based on GPS error or positioning error range; the fine matching clustering module is used to calculate the event feature similarity and / or environmental feature similarity between abnormal events, and cluster the abnormal events into the same event under the condition that the similarity meets the threshold θ and the spatial distance is less than the threshold Δd, and output a unique event identifier and its final localization result.

[0033] In a specific embodiment, the backend management platform includes a closed-loop processing module, which is used to generate processing work orders for deduplicated abnormal events and dispatch them, receive processing feedback and associate it with the corresponding unique event identifier, so as to form a closed-loop record of discovery, confirmation, processing and feedback, and output statistical evaluation results.

[0034] This application utilizes a ground inspection vehicle and an aerial drone to simultaneously inspect and collect data on the road section to be inspected. At both the ground and air ends, an AI-based edge reasoning model is used to output a set of candidate anomaly events. Then, time window alignment and coarse localization filtering under a unified time reference are used to achieve alignment and matching of the candidate events. Two-way verification is performed using anomaly category consistency, coarse localization distance threshold, and event feature vector similarity threshold as criteria, thus transforming the identification results from single-end judgment into mutually verifiable, highly reliable anomaly events. Based on this, the mutually verified anomaly events are used as calibration samples to perform consistency correction on the pixel-to-actual-scale conversion coefficient and compensate for the positioning deviation between airborne and vehicle-mounted systems. This achieves quantitative measurement parameters such as crack width / length, pothole area, and defect ratio, and is consistent with the positioning. Synchronous calibration of parameters, and then, taking into account the positioning error range, establishing a candidate set of spatial neighborhoods through coarse registration, and then combining fine matching and clustering of the abnormal event's own characteristics and environmental characteristics to complete the deduplication of duplicate events and the final positioning output, avoids repeated reporting and inaccurate landing points of the same abnormality due to GPS drift or viewpoint differences, significantly improving the measurability, locationability and dispatchability of inspection results; at the same time, by collaboratively planning air and ground inspection paths and setting lateral blind spot filling routes and circumferential verification routes to obtain multi-view airborne data, and generating work orders for deduplicated abnormal events in the background management platform and associating feedback to form a closed-loop record, the efficiency of inspection coverage and verification can be further improved, supporting the closed-loop management of traffic facilities and road surface defects from discovery, confirmation to handling feedback and statistical evaluation. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention.

[0037] Figure 2 This is a schematic diagram of the system structure of Embodiment 2 of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] The present invention will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to explain the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make equivalent substitutions or modifications to the parameter values, isolation action combinations, execution point types, etc., without departing from the concept of the present invention, and all such substitutions or modifications should fall within the scope of protection of the present invention.

[0040] Example 1: Intelligent Inspection Method for Integrated Air-Ground Transportation Facilities Based on Large Model

[0041] This embodiment focuses on a 10km long one-way road section from K12+000 to K22+000. The inspection includes identifying road surface cracks, potholes, repair marks, damaged road markings, abnormal guardrails, and debris. The inspection is conducted collaboratively by a ground inspection vehicle and an aerial drone, both equipped with edge computing capabilities and communicating with the backend management platform.

[0042] S1. Obtain patrol tasks and identify road sections to be patrolled.

[0043] The back-end management platform issues patrol tasks, including the start and end station numbers of the road section to be patrolled, the type of patrol object, the patrol time period, and the verification strategy parameters. To facilitate scheduling and alignment, the road section to be patrolled can be divided into multiple task segments according to the mileage range. The segment length can be 200-1000m. In this embodiment, 500m is used to form 20 task segments, and a ground inspection vehicle driving path and a drone flight path are assigned to each task segment.

[0044] When collaboratively planning air-to-ground inspection routes, the flight paths of drones include at least the following two types of routes:

[0045] (1) Lateral blind spot filling flight path: The UAV is offset laterally by 10-30m relative to the center line of the road, and the flight path is consistent with that of the inspection vehicle. It is used to fill the blind spot of the ground view and enhance the lateral observation of objects such as guardrails, isolation facilities, and markings. In this embodiment, the lateral offset is 15m.

[0046] (2) Circling verification route: When there is a suspected abnormal position, the UAV performs a circling shot with the suspected abnormal coarse positioning as the center. The circling radius can be 10-50m and the circling angular velocity can be 10-45° / s to obtain multi-view airborne acquisition data to support mutual verification and metrological calibration; in this embodiment, a radius of 20m is used for 1 circling.

[0047] S2 synchronously collects and binds time and location-related information.

[0048] The ground inspection vehicle is equipped with a forward-facing camera and a near-ground-viewing camera for data acquisition. The forward-facing camera has a resolution of 1920×1080 to 3840×2160 and a frame rate of 20 to 60fps; this embodiment uses 3840×2160 (4K) at 30fps. The inspection vehicle's speed can be 30 to 80 km / h; in this embodiment, it is 50 to 60 km / h. The near-ground-viewing camera can be installed at a height of 1.8 to 2.6m to enhance the acquisition of detailed textures such as cracks and potholes; in this embodiment, it is installed at 2.2m.

[0049] During the cruise phase, the UAV can fly at an altitude of 20–60 m, and the gimbal pitch angle can be -45° to -85°; in this embodiment, the cruise altitude is 30 m and the gimbal pitch angle is -70°. When a verification is triggered, the UAV can reduce its altitude to 10–25 m to improve spatial resolution; in this embodiment, the verification altitude is 15 m.

[0050] To support subsequent alignment and matching, both vehicle-mounted and airborne data are bound to metadata at the frame level, including at least:

[0051] (1) Time information: timestamp (can be in milliseconds) and frame number;

[0052] (2) Location-related information: Global positioning coordinates (latitude and longitude), and at least one of flight altitude, heading angle, pitch angle and roll angle.

[0053] The ground inspection vehicle and the drone align their time bases through a unified time synchronization mechanism or clock synchronization mechanism. The time synchronization method can be GNSS time synchronization, Network Time Protocol (NTP / PTP) time synchronization, or a combination of both, so that the timestamp deviation between the two ends does not exceed 5-50ms; in this embodiment, the timestamp deviation is controlled to ≤10ms.

[0054] S3 performs edge reasoning at both ends of the airspace and outputs a set of candidate abnormal events.

[0055] The ground inspection vehicle and drone terminals perform edge inference on the vehicle-collected data and airborne data, respectively, based on artificial intelligence recognition models, and output a set of candidate abnormal events. The artificial intelligence recognition model can be a "recognition model built based on the capabilities of a large model," such as a combination of a lightweight edge model obtained by distilling a large model and a cloud-based large model verification model. In this embodiment, lightweight edge models are deployed on the vehicle and airborne terminals for real-time inference, while the cloud is used for result aggregation and work order generation.

[0056] Edge inference can be batched by time slice, for example, forming an inference batch every 0.5 to 2 seconds and outputting candidate abnormal events within that time slice; in this embodiment, 1 second / batch is used. To reduce false positives, a confidence threshold can be set to filter candidate abnormal events, and the threshold can be 0.50 to 0.80; in this embodiment, 0.60 is used for the vehicle-mounted terminal and 0.55 is used for the drone terminal.

[0057] Each candidate anomaly event should include at least: anomaly category, confidence level, target area information, data collection timestamp, coarse location information, and event feature vector. For ease of engineering implementation, it can be organized into the following fields:

[0058] Anomaly categories: cracks, potholes, missing road markings, abnormal guardrails, spilled objects, etc.

[0059] Confidence level: 0 to 1;

[0060] Target region information: bounding box (bbox) and / or segmentation mask;

[0061] Collect timestamps and frame numbers;

[0062] Coarse positioning information: latitude and longitude coordinates and / or mileage marker;

[0063] Event feature vector: A representation vector used for cross-view matching and deduplication aggregation, with dimensions ranging from 64 to 1024; this embodiment uses 256 dimensions.

[0064] The event feature vector can be composed of two parts:

[0065] (1) Model intermediate features: These are obtained by the output of the intermediate feature layer corresponding to the target region by the artificial intelligence recognition model. For example, a fixed-dimensional vector is obtained by pooling the feature map of the second-to-last layer of the model with ROI.

[0066] (2) Explicit features: These are obtained by performing feature extraction on the target region and include at least morphological and / or texture features. Examples of morphological features include target area, perimeter, aspect ratio, skeleton length, number of crack branches, etc.; examples of texture features include edge density, gray-level co-occurrence statistics, histogram of oriented gradients (HOG), etc. Explicit features can be concatenated with or weighted and fused with intermediate features of the model to form the final event feature vector.

[0067] S4 Spatiotemporal Alignment Matching and Bidirectional Recognition Verification

[0068] Based on time and location information, the candidate abnormal events output by the ground inspection vehicle and the candidate abnormal events output by the drone are aligned and matched, and two-way recognition and verification are performed to determine the abnormal events that pass mutual verification.

[0069] In an optional implementation, alignment matching and bidirectional recognition verification include at least the following rules:

[0070] (1) Filter candidate pairings within a preset time window Δt. Δt can be 0.2 to 2.0 s; in this embodiment, Δt = 0.5 s is used. For vehicle-mounted candidate events, candidate pairings that satisfy |tc-tu| ≤ Δt are filtered from the UAV candidate event set;

[0071] (2) Coarse localization distance threshold Δd: Feature similarity determination is only performed when the coarse localization distance between candidate pairs is less than Δd. Δd can be 3 to 20 m; in this embodiment, Δd = 8 m is used.

[0072] (3) Feature similarity threshold θ: When the anomaly categories are consistent and the similarity of the event feature vectors is greater than θ, the candidate pairing is determined as a mutual verification passing anomaly event. The similarity calculation can be performed using cosine similarity, Euclidean distance transformed similarity, etc.; θ can be 0.75 to 0.95; in this embodiment, cosine similarity is used and θ=0.85 is taken.

[0073] When a candidate pair does not meet the mutual verification conditions, it is identified as an abnormal event pending review, and a review strategy is triggered. The review strategy includes at least one of the following:

[0074] (1) Instruct the UAV to perform supplementary photography at the location of the abnormal event to be reviewed (e.g., reduce the altitude to 15m and take pictures along the review route).

[0075] (2) Instruct the inspection vehicle to reduce its speed to 10-40 km / h and increase the sampling density at the location to be reviewed (for example, increase the key frame upload frequency to 3-10 fps); in this embodiment, the vehicle speed is reduced to 20-30 km / h and the key frame upload frequency is increased to 5 fps.

[0076] S5 performs calibration and metrology based on mutual certification and abnormal events.

[0077] For abnormal events that have passed mutual verification, calibration measurement shall be performed, and at least the quantitative measurement parameters and location parameters of the abnormal event shall be corrected.

[0078] (1) Quantitative metrological calibration

[0079] For objects requiring quantification, such as cracks, potholes, and missing road markings, pixel-to-actual-scale conversion coefficients are calculated on both the vehicle-mounted and airborne ends based on camera intrinsic parameters, attitude information, and shooting height information. The conversion coefficients can be expressed as kkk (unit: mm / px) or as a two-dimensional scaling factor. To reduce measurement bias introduced by differences in viewpoint and height, cross-verification failure events are used as calibration samples to perform consistency correction on the vehicle-mounted conversion coefficient kck_ckc and the airborne conversion coefficient kuk_uku, for example, using weighted fusion.

[0080] ;

[0081] In this embodiment, α = 0.5 is used. Subsequently, the width, length or area at the pixel scale is calculated based on the target region information (e.g., crack mask), and then combined with the corrected conversion coefficient to output the quantization measurement parameters.

[0082] For example, the vehicle-mounted device calculates the maximum width pixels for a certain crack event. Vehicle conversion factor ; Calculation after drone verification After fusion Then output the maximum width of the crack. The pit area can be calculated by multiplying the pixel area by k. 2 The conversion can be performed; the defect ratio can be calculated as the defect area / the baseline area.

[0083] (2) Positioning calibration mutual verification utilizes both on-board and airborne coarse positioning through abnormal events. The deviation vector between the two can be calculated. (This can be represented in a meter-level planar coordinate system), and the deviation is used to compensate for the deviation in the coarse positioning information of subsequent abnormal events within the same task segment, so as to reduce the final positioning drift. The deviation compensation can be updated by time slice or mileage segment. In this embodiment, the deviation compensation amount is updated at least once within each task segment.

[0084] S6 performs coarse registration and fine matching, deduplication and aggregation, and outputs the final localization result.

[0085] To handle positioning errors and duplicate reporting, coarse registration is performed on abnormal events, followed by fine matching, deduplication, and aggregation.

[0086] Coarse registration establishes a candidate set of spatial neighbors based on the positioning error range. The positioning error range can be set according to GPS error statistics or experience, and the radius R can be 8-30m; in this embodiment, R=12m is used. When the final positioning distance between two abnormal events does not exceed R, fine matching begins.

[0087] Fine-grained matching is based at least on the characteristics of the anomalous event itself and / or the characteristics of the environment in which the anomalous event occurs:

[0088] Self-characteristics: The similarity of event feature vectors is used as the criterion, and the threshold can be 0.80 to 0.95; in this embodiment, 0.88 is used.

[0089] Environmental characteristics: Extract environmental anchor points around the abnormal event and calculate their similarity. Environmental anchor points may include lane line segmentation results, guardrail segment detection results, sign detection frames and their relative geometric relationship with the abnormal target, curb lines / fence outlines, etc. The environmental similarity threshold can be 0.70 to 0.90; in this embodiment, it is set to 0.80.

[0090] Abnormal events that meet the criteria of "spatial distance ≤ R and (self-feature similarity ≥ threshold or environmental similarity ≥ threshold)" are aggregated into a single event, generating a unique event identifier (event_id), and their locations are then fused and output. The fused location can be obtained by weighting the mutual verification confidence level to obtain the final location, while retaining the mutual verification evidence chain (air-ground keyframes, similarity values, correction coefficients, deviation compensation amounts, etc.) to facilitate subsequent order dispatching and verification.

[0091] S7 generates inspection results and outputs them to the backend management platform.

[0092] The deduplicated anomalies and their final location results are output to the backend management platform. The platform generates an inspection record and a handling work order for each event_id. Work order fields may include: event type, location (latitude and longitude / station number), quantitative measurement parameters (such as maximum crack width, pit area, and defect ratio), mutual verification confidence level, evidence image / video index, and recommended handling level. After handling, the "before / after" information is filled in and associated with the event_id, forming a closed-loop record of discovery—confirmation—handling—feedback, and outputting statistical evaluation results.

[0093] Example 2: Intelligent Inspection System for Integrated Air-Ground Transportation Facilities Based on Large Model

[0094] The system in this embodiment is used to execute the method described in Embodiment 1, and includes: an air-ground collaborative acquisition unit, a spatiotemporal alignment unit, an edge inference unit, a mutual verification judgment unit, a calibration and measurement unit, a deduplication and positioning unit, a result output unit, and a background management platform.

[0095] (1) Air-ground collaborative acquisition unit

[0096] The air-to-ground collaborative data acquisition unit comprises a vehicle-mounted acquisition subunit and a drone acquisition subunit. The vehicle-mounted acquisition subunit includes camera equipment and a positioning / attitude acquisition module, used to acquire vehicle-mounted video and generate metadata such as timestamps, frame numbers, and latitude / longitude coordinates. The drone acquisition subunit includes a gimbal camera and a flight control output interface, used to acquire airborne video and generate metadata such as timestamps, frame numbers, latitude / longitude coordinates, flight altitude, heading angle, pitch angle, and roll angle. The air-to-ground collaborative acquisition unit binds the acquired data and metadata and transmits them to the subsequent processing unit.

[0097] (2) Spatiotemporal alignment unit

[0098] The spatiotemporal alignment unit is used to align time windows of data collected from both the air and ground ends based on a unified time reference, and to perform candidate pairing filtering based on coarse positioning information. In an optional implementation, the spatiotemporal alignment unit includes:

[0099] The clock synchronization module is used to ensure that the timestamps of the vehicle-mounted terminal and the drone terminal meet the same time reference, with the deviation controlled within 5 to 50 ms;

[0100] The time window index module is used to establish a correspondence and output a candidate pairing set with a preset time window Δt, where Δt can be 0.2 to 2.0 s; in this embodiment, it is 0.5 s. At the same time, candidate pairings are filtered based on a coarse positioning distance threshold Δd, where Δd can be 3 to 20 m; in this embodiment, it is 8 m.

[0101] (3) Marginal reasoning unit

[0102] The edge inference unit is deployed on both the vehicle-mounted and drone-mounted terminals. It performs edge inference on the collected data based on an AI-powered recognition model, outputting a set of candidate anomaly events. Each candidate anomaly event includes at least anomaly category, confidence level, target region information, timestamp, coarse localization information, and event feature vector. The event feature vector can be obtained by fusing intermediate layer features from the model with explicit morphological / texture features, used for cross-view matching and deduplication aggregation. The edge inference unit can output results in time slices, for example, every 0.5–2 seconds; in this embodiment, it outputs every 1 second.

[0103] (4) Mutual verification determination unit

[0104] The mutual verification determination unit is used to perform category consistency determination and event feature similarity calculation on candidate pairs, and outputs mutual verification passed abnormal events and pending verification abnormal events based on preset thresholds. The mutual verification criteria include: category consistency, coarse positioning distance less than Δd, and event feature similarity greater than θ; θ can be 0.75 to 0.95, and is taken as 0.85 in this embodiment. The mutual verification determination unit can send a verification command to the acquisition control terminal to trigger UAV re-shooting or inspection vehicle deceleration and sampling.

[0105] (5) Calibration of the metrology unit

[0106] The calibration metrology unit is used to correct quantitative metrology parameters and / or positioning parameters by using cross-certification failure events as calibration samples.

[0107] The quantization calibration module calculates the pixel-to-actual-scale conversion coefficient based on camera intrinsic parameters, attitude information, and shooting height information, and performs consistency correction using mutual verification samples, outputting at least one of the following: crack width, crack length, pit area, and defect ratio; the positioning calibration module is used to calculate the deviation vector between computer-based coarse positioning and vehicle-based coarse positioning, and performs deviation compensation for subsequent event coarse positioning to reduce final positioning drift.

[0108] (6) De-positioning unit

[0109] The deduplication localization unit is used to perform coarse registration based on the localization error range, and further perform fine matching based on the characteristics of the abnormal event itself and / or the characteristics of the environment in which the abnormal event is located, to deduplicate and aggregate duplicate abnormal events and output the final localization result.

[0110] In an optional implementation, the deduplication localization unit includes a coarse registration module and a fine matching clustering module:

[0111] The coarse registration module establishes a candidate set of spatial neighborhoods based on GPS error or positioning error range, with a radius R that can be 8 to 30 m; in this embodiment, it is 12 m.

[0112] The fine-grained matching clustering module calculates the similarity of event features and / or environmental features. When the similarity meets a threshold and the spatial distance is less than a preset range, abnormal events are clustered into the same event, outputting a unique event identifier (event_id) and its final location result. Environmental features can consist of lane lines, guardrail segments, signage targets, and their relative relationships.

[0113] (7) Result output unit and back-end management platform

[0114] The results output unit generates inspection results and outputs them to the backend management platform. The backend management platform includes a closed-loop handling module, which generates and dispatches handling work orders for deduplicated abnormal events, receives handling feedback and associates it with the event_id to form a closed-loop record, and outputs statistical evaluation results. Statistical content may include the quantity distribution and trends by road segment, type, and severity, handling timeliness, and recurrence rate, providing a basis for maintenance decisions.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent inspection of integrated air-ground transportation facilities based on a large model, characterized in that, include: S1. Obtain patrol tasks and determine the road sections to be patrolled; S2. Simultaneously patrol and collect data on the road section to be inspected using ground inspection vehicles and aerial drones, obtain vehicle-mounted data and airborne data, and bind time information and location-related information to the vehicle-mounted data and airborne data respectively. S3. On the ground inspection vehicle and the airborne drone, respectively, edge reasoning is performed on the vehicle-mounted data and airborne data based on the artificial intelligence recognition model to output a candidate abnormal event set. The candidate abnormal event set includes at least the category information of traffic facility abnormalities or road surface defects and the corresponding target area information. S4. Based on the time information and the location-related information, the candidate abnormal events output by the ground inspection vehicle terminal and the candidate abnormal events output by the aerial drone terminal are aligned and matched, and bidirectional identification and verification are performed to determine the abnormal events that have passed mutual verification. S5. Perform calibration measurement based on the abnormal event of the mutual verification, and at least correct the quantitative measurement parameters and positioning parameters of the abnormal event; S6. Perform coarse registration on abnormal events based on the positioning error range, and further perform fine matching based on the characteristics of the abnormal events themselves and the characteristics of the environment in which the abnormal events are located, so as to deduplicatize and aggregate duplicate abnormal events, and output the deduplicated abnormal events and their final positioning results. S7. Generate inspection results and output them to the backend management platform.

2. The method according to claim 1, characterized in that, When S1 coordinates the planning of the inspection path of the ground inspection vehicle and the inspection path of the aerial drone, it includes at least: dividing the inspection task into multiple task segments according to the mileage range of the road segment to be inspected; assigning the driving path of the ground inspection vehicle and the flight path of the aerial drone to each task segment; and setting a lateral blind spot filling route and a circling verification route around the suspected abnormal location for the aerial drone related to the driving direction of the ground inspection vehicle, so as to obtain airborne acquisition data from different perspectives.

3. The method according to claim 1, characterized in that, The time and location information bound to the vehicle-mounted and airborne data in S2 includes at least: timestamp, frame number; and at least one of global positioning coordinates, flight altitude, heading angle, pitch angle, and roll angle; and the ground inspection vehicle and the airborne UAV end use a unified time synchronization mechanism to ensure that the timestamps of both ends fall into the same time base for alignment and matching in S4.

4. The method according to claim 1, characterized in that, Each candidate abnormal event output by S3 includes at least: abnormal category, confidence level, target area information, collection timestamp, coarse location information, and event feature vector; The event feature vector is obtained by the output of the intermediate feature layer corresponding to the target region by the artificial intelligence recognition model and by performing feature extraction on the target region; the feature extraction includes at least morphological features and / or texture features.

5. The method according to claim 1, characterized in that, The alignment matching and bidirectional recognition verification of S4 includes at least the following: within a preset time window Δt, screening candidate pairs of vehicle-mounted candidate abnormal events and airborne candidate abnormal events; for each candidate pair, if the abnormal categories are consistent and the coarse positioning distance between the two is less than a preset distance threshold Δd, then calculating the similarity of the event feature vectors; when the similarity is greater than a preset threshold θ, the candidate pair is determined as an abnormal event that has passed mutual verification; otherwise, the candidate pair is determined as an abnormal event to be reviewed and a review strategy is triggered; wherein, the review strategy includes at least one of the following: instructing the aerial UAV to perform supplementary image acquisition at the location of the abnormal event to be reviewed and / or instructing the ground inspection vehicle to reduce its speed and increase the sampling density at the location of the abnormal event to be reviewed.

6. The method according to claim 1, characterized in that, The calibration and measurement of S5 includes at least one of quantitative measurement calibration and positioning calibration. Quantitative measurement calibration includes: for mutually verified abnormal events, calculating pixel-to-actual-scale conversion coefficients on both the vehicle-mounted and airborne ends based on camera intrinsic parameters, attitude information, and shooting height information, and performing consistency correction on the conversion coefficients based on the mutually verified abnormal events to obtain at least one of the corrected quantitative measurement parameters, such as crack width, crack length, pit area, and defect ratio. Positioning calibration includes: for mutually verified abnormal events, calculating the deviation vector between computer-based coarse positioning and vehicle-mounted coarse positioning, and performing deviation compensation on the coarse positioning information of subsequent abnormal events to reduce final positioning drift.

7. A large-scale integrated air-ground intelligent inspection system for transportation facilities, characterized in that, include: The air-ground collaborative data acquisition unit is used for simultaneous inspection and data collection of the road section to be inspected by ground inspection vehicles and airborne drones, to obtain vehicle-mounted data and airborne data, and to bind time information and location-related information to the vehicle-mounted data and airborne data. The spatiotemporal alignment unit is used to align the time windows of data collected at both ends of the air and ground based on a unified time reference, and to perform candidate pairing and filtering based on coarse positioning information; The edge reasoning unit is used to perform edge reasoning on the collected data based on artificial intelligence recognition models at both the ground inspection vehicle end and the air drone end, and output a set of candidate abnormal events. Each candidate anomaly event includes at least the anomaly category, confidence level, target area information, timestamp, coarse localization information, and event feature vector; The mutual verification determination unit is used to perform category consistency determination and event feature similarity calculation on candidate pairings, and output mutual verification pass abnormal events and pending verification abnormal events based on preset thresholds; The calibration metrology unit is used to correct quantitative metrology parameters and / or positioning parameters by using cross-certification failure events as calibration samples. The deduplication localization unit is used to perform coarse registration based on the localization error range, and further perform fine matching based on the characteristics of the abnormal event itself and / or the characteristics of the environment where the abnormal event is located, to deduplicate and aggregate duplicate abnormal events and output the final localization result; The results output unit is used to generate inspection results and output them to the backend management platform.

8. The system according to claim 7, characterized in that, The spatiotemporal alignment unit includes a clock synchronization module and a time window index module; the clock synchronization module is used to ensure that the timestamps of the ground inspection vehicle and the airborne UAV meet a unified time reference; the time window index module is used to establish a correspondence between the vehicle-mounted data and the airborne data with a preset time window Δt, so as to support the candidate pairing screening and mutual verification judgment.

9. The system according to claim 7, characterized in that, The deduplication localization unit includes a coarse registration module and a fine matching clustering module: the coarse registration module is used to establish a spatial neighborhood candidate set for abnormal events based on GPS error or positioning error range; the fine matching clustering module is used to calculate the event feature similarity and / or environmental feature similarity between abnormal events, and cluster abnormal events into the same event under the condition that the similarity meets the threshold θ and the spatial distance is less than the threshold Δd, and output a unique event identifier and its final localization result.

10. The system according to claim 7, characterized in that, The back-end management platform includes a closed-loop processing module, which generates processing work orders for deduplicated abnormal events and dispatches them, receives processing feedback and associates it with the corresponding unique event identifier to form a closed-loop record of discovery, confirmation, processing and feedback, and outputs statistical evaluation results.

Images