A multi-source perception-based highway three-dimensional intelligent inspection system and method

Abstract

This invention discloses a multi-source perception-based intelligent highway inspection system and method, specifically relating to the field of intelligent traffic inspection technology, to address the problem of low reliability in multi-source data fusion on highways. The invention collects multi-source data by deploying cameras, thermal imaging devices, and vehicle or airborne platforms in highway scenarios; it executes Boolean consistency gating and mutual exclusion rejection within a unified time window, cross-validates to generate highly reliable candidates, solves cross-modal extrinsic parameters using road geometric anchor points and a stochastic consistency model, and uses threshold rollback for extrinsic parameter change rates to ensure long-term stable alignment. Then, based on an inspection anomaly truth table, it confirms foreign objects, water accumulation, smoke, and missing road markings using necessary and sufficient conditions, and tracks the evolution using a finite state machine. The confirmed upgrade node schedules vehicle and airborne platforms and links with on-site facilities based on the shortest path of the topological graph, forming a closed loop from discovery to handling. This significantly improves efficiency and reliability, reduces costs and misjudgments, and possesses advantages of intelligence, globalization, and adaptability.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic inspection technology, and more specifically, to a three-dimensional intelligent highway inspection system and method based on multi-source perception. Background Technology

[0002] As the scale of the highway network continues to expand and the intensity of traffic operation continues to increase, the demand for safe operation and maintenance and risk prevention and control of road infrastructure is increasing. In order to achieve efficient detection and handling of abnormal situations such as road surface structure status, integrity of signs and markings, obstruction of roads by foreign objects, water accumulation and smoke, more and more highway and urban road management systems are beginning to introduce intelligent inspection methods based on artificial intelligence. As a joint inspection system that integrates multiple platforms such as aerial drones, ground unmanned vehicles and roadside fixed equipment, the three-dimensional intelligent inspection system has multi-angle and multi-level perception coverage capabilities and can realize target recognition, status assessment and task collaborative execution in dynamic environments.

[0003] The shortcomings of existing technologies are as follows: In existing highway inspection systems, multiple sensing devices such as cameras, millimeter-wave radar, lidar, and thermal imaging devices are generally deployed and operated independently, lacking a unified time synchronization and spatial registration mechanism. This leads to problems such as asynchronous acquisition of sensing data, spatial mapping deviation, and conflicting recognition results. At the same time, multimodal fusion mostly relies on static thresholds or scoring models, lacking dynamic adaptability to complex inspection environments (such as tunnel entrances and exits, obstructed areas, strong light interference, etc.), which easily produces false alarms or missed alarms. Moreover, the identification and hierarchical management of inspection targets often adopt single judgment or end-to-end output, lacking state evolution models and evidence chain tracking mechanisms, making it difficult to accurately capture the development trend of events. In terms of task linkage and platform collaboration, there is a lack of spatial topology perception and priority scheduling capabilities, making it impossible to achieve task relay and intelligent orchestration among drones, ground vehicles, and roadside equipment, resulting in low efficiency of inspection resource utilization and untimely closed-loop feedback. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a highway three-dimensional intelligent inspection system and method based on multi-source perception, so as to solve the problem of low reliability of multi-source data fusion in the above-mentioned background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for intelligent three-dimensional highway inspection based on multi-source sensing includes the following steps:

[0007] Based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios, Boolean consistency gating is performed on multimodal outputs falling into the same grid within the synchronization time window, and mutual exclusion rules are applied to generate candidate events.

[0008] Candidate events are spatiotemporally aligned and fused. Time cross-correlation is used for timestamp alignment. Cross-modal extrinsic parameters are solved based on the stochastic consistency model of road geometric anchor points. An extrinsic parameter change rate threshold is set to trigger extrinsic parameter rollback and complete coordinate unification.

[0009] Based on the established inspection anomaly truth table rules, the validity of events is confirmed by combining coarse screening features and category features. A set of valid events is output, and results that do not meet the sufficient conditions or are mutually exclusive are marked as review candidates.

[0010] A finite state machine is constructed to manage events hierarchically. State transitions are performed based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and inspection records are generated.

[0011] Based on the constructed topology map, multi-platform collaborative scheduling is performed. The reachable scheduling path with the shortest path length is selected, and the vehicle-mounted or airborne platform is used for verification and re-inspection. The on-site facilities are also linked to complete the handling and response.

[0012] In areas with limited positioning signals, path calibration and event location alignment are performed, and inspection strategies and rule entries are updated in conjunction with historical inspection data to achieve dynamic optimization.

[0013] In a preferred embodiment, Boolean consistency gating is performed on multimodal outputs falling into the same grid within the synchronization time window, and a mutual exclusion rule is applied to generate candidate events. The specific process is as follows:

[0014] Under a unified time reference, a sliding time window is set to perform time-series alignment of abnormal target information collected by different types of sensing terminals, and grid areas are divided based on the road spatial structure to map multimodal sensing results with spatiotemporal labels to the corresponding spatial grid units.

[0015] When the same grid cell receives anomaly category labels from different modalities within the same time window, it is judged according to the set Boolean consistency gating condition. If at least two different modal devices trigger detection results for the same anomaly category within the time window, and the time overlap ratio meets the set threshold, the anomaly is determined to be a consistent trigger and enters the candidate event list.

[0016] Under the premise that Boolean consistency is true, further filtering is performed by applying the mutual exclusion rule, which is a combination relationship where there is a physical or logical contradiction between different modalities;

[0017] When mutually exclusive combinations exist, the event triggering result is either rejected or downgraded and marked as a review candidate.

[0018] Events that satisfy Boolean consistency gating and are not mutually exclusive are output as valid candidate events, and their corresponding grid positions, anomaly categories, trigger mode types, and time intervals are recorded.

[0019] In a preferred embodiment, the mutually exclusive rejection rules include visually identifying a closed contour but the radar echo is empty, visually detecting smoke texture but thermal imaging showing no temperature diffusion characteristics, and thermal imaging showing cold or hot anomalies but neither visual nor radar continuously detecting the object.

[0020] In a preferred embodiment, timestamp alignment using time cross-correlation refers to alignment using the cross-current cross-correlation peak positioning method. The threshold for the rate of change of external parameters is set according to the equipment structure and the upper limit of thermal drift. When the threshold is continuously exceeded, external parameter rollback and data reprojection are performed.

[0021] In a preferred embodiment, cross-modal extrinsic parameters are solved based on a stochastic consistency model of road geometric anchor points, and an extrinsic parameter change rate threshold is set to trigger extrinsic parameter rollback, thereby completing coordinate unification. The specific process is as follows:

[0022] The spatial structural features deployed in the inspection area are used as geometric anchors, including road mileage markers, tunnel portal outlines, road edge boundaries, niche structures, traffic sign corners and intersections of road markings;

[0023] Based on the observation results of geometric anchor points in different modes, a multimodal observation aligned sample set is constructed. The random sampling consistency model is used to iteratively fit the sample set and solve the spatial extrinsic transformation matrix between each mode at the current time.

[0024] During continuous operation, the historical change trajectory of the external parameters is recorded, and the rate of change of the external parameters is calculated. The rate of change of the external parameters is the change amplitude of the rotation and translation components in the external parameter matrix per unit time.

[0025] When the rate of change of the extrinsic parameters exceeds the set threshold and the parameters are in an abnormal drift state for multiple consecutive time periods, the current fusion alignment state is determined to be unreliable, triggering the extrinsic parameter rollback mechanism. The current extrinsic parameters are reset to the extrinsic parameter matrix that was previously marked as stable, and the writing of the fusion results for this round is stopped.

[0026] In a preferred embodiment, the inspection anomaly truth table specifies minimum sufficient condition combinations for foreign objects, water accumulation, smoke, and marking defects. Among them, foreign object confirmation satisfies either a visual closed contour and stable radar echo or a visual continuous satisfaction and thermal image cold and hot anomalies. Mutually exclusive candidate events are automatically downgraded to review candidates.

[0027] In a preferred embodiment, a finite state machine is constructed to manage events hierarchically. State transitions are performed based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and inspection records are generated. The specific process is as follows:

[0028] Define the event state set as including undetermined state, confirmed state, escalation state and recovery state, and construct a finite state machine with explicit state transition paths to describe the life cycle evolution process of the event;

[0029] Events in the pending state are connected to a finite state machine. When the event satisfies the consistency triggering condition and the effective duration within the continuous observation window is greater than the preset duration threshold, it transitions from the pending state to the confirmed state.

[0030] After an event enters the confirmed state, if the spatial coverage of the abnormal target shows a monotonically increasing trend in a continuous time window, or if the abnormality intensity index continues to increase, the transition from the confirmed state to the escalation state is triggered.

[0031] If, after an event enters the escalation or confirmation state, no abnormal target is detected again within a set number of consecutive time windows, the event transitions from the current state to the recovery state and is marked as pending clearance or completed revisit.

[0032] During the event state change process, the entry time, exit time, state transition path, and criteria for triggering the transition for each state are recorded, and structured inspection records are generated simultaneously.

[0033] In a preferred embodiment, multi-platform collaborative scheduling is performed based on the constructed topology map. The reachable scheduling path with the shortest path length is selected for on-board or airborne platform verification and re-inspection, and on-site facilities are coordinated to complete the response. The specific process is as follows:

[0034] Based on the road structure, monitoring equipment deployment location and platform dwell point information within the inspection area, a topology graph is constructed. The topology graph contains nodes and edges. Nodes represent the spatial location of vehicle-mounted platforms, airborne platforms or field equipment, and edges represent the traffic reachable paths between nodes and their distance or cost attributes.

[0035] Once an event enters the confirmed or escalated state, based on a comprehensive assessment of the event location and platform reachability, the shortest path is calculated using a topology graph, prioritizing the scheduling path with the shortest path length and available platform status.

[0036] Depending on the platform type, vehicle-mounted or airborne platforms may be dispatched to perform certification or re-inspection tasks.

[0037] Simultaneously, the on-site facilities deployed near the affected area are coordinated, and response actions are controlled and issued according to the event type and the corresponding handling template.

[0038] A multi-source sensing-based intelligent highway inspection system, used to implement the aforementioned multi-source sensing-based intelligent highway inspection method, includes:

[0039] The candidate event generation module is used to generate candidate events based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios. Within the synchronization time window, it performs Boolean consistency gating on multimodal outputs falling into the same grid and applies mutual exclusion rules.

[0040] The event analysis module performs spatiotemporal alignment and fusion processing on candidate events, uses time cross-correlation for timestamp alignment, solves cross-modal extrinsic parameters based on the stochastic consistency model of road geometric anchor points, and sets a threshold for the rate of change of extrinsic parameters to trigger extrinsic parameter rollback, thus completing coordinate unification.

[0041] The anomaly review module performs event validity verification based on the set inspection anomaly truth table rules and combines coarse screening features and category features. It outputs a set of valid events and marks results that do not meet sufficient conditions or are mutually exclusive as review candidates.

[0042] The inspection hierarchical management module constructs a finite state machine to manage events hierarchically. It performs state transitions based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and generates inspection records.

[0043] The dispatch and handling module performs multi-platform collaborative dispatch based on the constructed topology map, selects the reachable dispatch path with the shortest path length, and conducts on-board or airborne platform verification and re-inspection, and coordinates with on-site facilities to complete the handling response;

[0044] The inspection strategy update module performs path calibration and event location alignment in areas with limited positioning signals, and updates the inspection strategy and rule entries based on historical inspection data to achieve dynamic optimization.

[0045] The technical effects and advantages of this invention are as follows:

[0046] This invention deploys camera equipment, radar devices, thermal imaging terminals, and vehicle-mounted and airborne platforms in highway inspection scenarios to collect multi-source sensing data. Within a unified time window, Boolean consistency gating and mutual exclusion rules are used to cross-validate the detection results of different modalities, eliminating logically contradictory triggering events and generating a highly reliable candidate event set, effectively improving the accuracy and reliability of anomaly identification. Furthermore, a stochastic consistency model is constructed based on road geometric anchor points to dynamically solve cross-modal extrinsic parameters. Combined with a rollback mechanism triggered by a threshold for the rate of change of extrinsic parameters, alignment distortion caused by equipment drift and thermal noise is avoided, ensuring the uniformity and long-term stability of multimodal data in spatial coordinates.

[0047] By combining the truth table rules of inspection anomalies, necessary and sufficient conditions are set for various anomalies such as foreign objects, water accumulation, smoke, and missing road markings, enabling automatic confirmation and hierarchical management of events. The life cycle evolution of events is tracked through a finite state machine, accurately reflecting the occurrence, escalation, and resolution of events. At key nodes of confirmation or escalation, multi-platform collaborative scheduling is performed based on the topology map, intelligently selecting the scheduling path with the shortest path length and reachability. The vehicle-mounted and airborne platforms are coordinated to efficiently complete evidence collection or re-inspection, and the on-site facilities are linked to trigger the response. This achieves closed-loop management of the entire chain from discovery to handling, thereby significantly improving inspection efficiency and reliability, reducing labor costs and misjudgment rate, and possessing the technical advantages of intelligence, globalization, and long-term self-adaptation. Attached Figure Description

[0048] Figure 1 This is a flowchart of a three-dimensional intelligent highway inspection method based on multi-source sensing according to the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of a three-dimensional intelligent highway inspection system based on multi-source sensing according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1: As Figure 1 As shown, a three-dimensional intelligent highway inspection method based on multi-source perception includes the following steps:

[0052] Based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios, Boolean consistency gating is performed on multimodal outputs falling into the same grid within the synchronization time window, and mutual exclusion rules are applied to generate candidate events.

[0053] Candidate events are spatiotemporally aligned and fused. Time cross-correlation is used for timestamp alignment. Cross-modal extrinsic parameters are solved based on the stochastic consistency model of road geometric anchor points. An extrinsic parameter change rate threshold is set to trigger extrinsic parameter rollback and complete coordinate unification.

[0054] Based on the established inspection anomaly truth table rules, the validity of events is confirmed by combining coarse screening features and category features. A set of valid events is output, and results that do not meet the sufficient conditions or are mutually exclusive are marked as review candidates.

[0055] A finite state machine is constructed to manage events hierarchically. State transitions are performed based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and inspection records are generated.

[0056] Based on the constructed topology map, multi-platform collaborative scheduling is performed. The reachable scheduling path with the shortest path length is selected, and the vehicle-mounted or airborne platform is used for verification and re-inspection. The on-site facilities are also linked to complete the handling and response.

[0057] In areas with limited positioning signals, path calibration and event location alignment are performed, and inspection strategies and rule entries are updated in conjunction with historical inspection data to achieve dynamic optimization.

[0058] Step 1: Based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, and vehicle-mounted and airborne platforms deployed in highway scenarios, Boolean consistency gating is performed on multimodal outputs falling into the same grid within the synchronization time window, and mutual exclusion rules are applied to generate candidate events. Specific steps include:

[0059] Set the sliding time window length parameter and window step size parameter, and establish spatial grid units based on the road centerline, lane width and device field of view. Each sensing output is accompanied by a time label and a spatial label: the time label is the start and end time of the acquisition, and the spatial label is the spatial grid unit number that falls into after being projected onto a unified coordinate system. When a sensing output covers multiple grid units, it is registered one by one according to the grid units it covers. The multimodal sensing results with spatiotemporal labels are mapped to the corresponding spatial grid units in each sliding time window.

[0060] Within the current sliding time window of any spatial grid cell, anomaly category labels and trigger time intervals from different modalities are aggregated. The anomaly category label is the semantic judgment result of the grid cell within the window, such as foreign objects, water accumulation, smoke, and missing markings. The trigger time interval is the start and end time of the modality continuously reporting the anomaly category within the window.

[0061] Boolean consistency gating is applied: when at least two different modalities trigger for the same anomaly category, and the overlap ratio between the trigger time intervals of the two modalities reaches the time overlap ratio threshold parameter, it is determined to be a consistent trigger. The time overlap ratio is calculated as follows: first, the intersection duration of the two trigger time intervals is taken, and then the ratio of this intersection duration to the shorter of the two trigger time intervals is compared. If the ratio is not less than the time overlap ratio threshold parameter, the time overlap condition is met.

[0062] Considering the slight spatial misalignment caused by the field of view boundaries of different devices, if the triggering results of the two modes fall into the target grid cell and its adjacent grid cell respectively and the coverage boundary is continuous, it is also considered as an effective spatial overlap falling into the same grid cell. The abnormal categories that meet the above gating conditions are added to the candidate event list, while those triggered by only a single mode or that do not reach the time overlap ratio threshold are not added.

[0063] After Boolean consistency is established, a set of mutually exclusive rules is applied to consistent trigger entries within the same grid cell and the same time window. The set of mutually exclusive rules defines the physical and logical contradictions between modalities and includes at least the following three types of rules: First, the visual modality identifies a closed contour while the radar modality has no stable echo; a closed contour refers to a closed boundary formed after segmentation or edge tracking with an area not less than a preset pixel area threshold, and maintaining inter-frame continuity within the window; a stable radar echo refers to the target distance and echo amplitude returned by multiple consecutive radar echo frames to the same grid cell within the window remaining within the allowable fluctuation range. If it does not exist at all or is only an isolated frame, it is considered as having no stable echo; Second, the visual modality detects smoke texture while the thermal imaging modality has no temperature diffusion characteristics. The process for determining smoke texture is as follows: Within the window, a decrease in local image contrast and a diffuse texture change are observed, exhibiting continuity in adjacent frames. The process for determining temperature diffusion characteristics is as follows: The thermal imaging temperature field shows a temperature gradient expansion from the central region outwards in adjacent frames. If only instantaneous high and low temperature noise appears without continuous diffusion, it is determined to have no temperature diffusion characteristics. Thirdly, if the thermal imaging mode exhibits cold and hot anomalies while neither the visual nor radar modes continuously detect objects, cold and hot anomalies refer to stable areas with temperatures higher or lower than the background temperature appearing continuously for several frames within the window. The failure to continuously detect objects refers to the visual and radar modes failing to conclude the existence of a target in the same grid cell within the window under the condition that the number of consecutive frames reaches the continuous frame threshold. If any mutually exclusive rule is hit, the consistent trigger entry is rejected, or it is downgraded and marked as a review candidate according to the strategy to enter the subsequent manual or secondary acquisition review process.

[0064] When an anomaly category passes the Boolean consistency gating within the current grid cell without being mutually exclusive, a valid candidate event is generated. The record of a valid candidate event includes at least: the spatial grid cell number and its center coordinates in a unified coordinate system, the anomaly category, a list of triggering modal types (clearly listing each modality participating in the consistency triggering), the triggering time interval (clearly specifying the start and end times), and the mutual exclusion check result (marked as passed or not triggered). To avoid duplicate entries caused by window sliding, the system merges valid candidate events with the same anomaly category and consecutive or contiguous triggering time intervals within adjacent time windows. The merged time interval takes the earliest start time and the latest end time, while the spatial location keeps the grid cell number unchanged, and the center coordinates are given according to the center of the grid cell. This merging process does not change the anomaly category and the list of triggering modal types; it only continuously aggregates the time dimension.

[0065] Example 1: Within a spatial grid cell on a straight road segment, the visual modality continuously outputs the foreign object category and forms a closed contour within the current sliding time window. The radar modality continuously transmits stable echoes back to the same grid cell within the same window. The intersection of their trigger time intervals reaches a preset time overlap ratio threshold compared to the shorter trigger interval length, and the Boolean consistency gating passes. The mutual exclusion rejection rule is not hit (because the radar has stable echoes and the thermal imaging has no negative evidence). Based on this, a valid candidate event (foreign object) is generated.

[0066] Example 2: In a spatial grid cell at the entrance of a ramp, the visual mode continuously reports the smoke type within the current window, while the thermal imaging mode does not show the characteristic of the temperature field spreading outward within the same window, and the radar mode also does not continuously detect the target echo. In this case, the mutually exclusive rejection rule of visual smoke and thermal imaging without temperature diffusion characteristics is applied. This consistent triggering entry is downgraded and marked as a review candidate, and is not output as a valid candidate event, pending subsequent secondary acquisition or manual review and confirmation.

[0067] Step 2 involves spatiotemporal alignment and fusion of candidate events. Time stamp alignment is achieved using temporal cross-correlation. Cross-modal extrinsic parameters are solved based on a stochastic consistency model of road geometric anchor points. An extrinsic parameter change rate threshold is set to trigger extrinsic parameter rollback, thus completing coordinate unification. Specific steps include:

[0068] First, the multimodal sensing link is run under a unified time reference. The sliding time window length and window step size parameters are set, and time-stamped multimodal sensing data streams are acquired. Timestamp alignment is performed using a cross-stream cross-correlation peak localization method. The specific steps are as follows:

[0069] For each sensing data stream, a time-series feature trajectory synchronized with the window is constructed. This trajectory can consist of stable and extractable statistics within each frame, such as binary pulses indicating the presence of a valid target frame, the number of geometric anchors detected in that frame, or a trackable intensity index for the same anchor. During the alignment phase, using the time axis of the visual data stream as a reference, the time-series feature trajectories of the radar data stream, lidar data stream, and thermal imaging data stream are subjected to slip and correlation calculations respectively.

[0070] The aligned data streams are gradually shifted forward or backward in time. When the overall similarity between two trajectories is highest at a certain shift amount, that shift amount is determined to be the time offset between the two trajectories. The highest overall similarity is measured by the difference between the number of consistent changes in the same direction and the number of inconsistent changes in opposite directions of the two trajectories in all overlapping frames. The shift amount with the largest difference is the location point of the cross-correlation peak. After determining the offset of each data stream relative to the reference time axis, its time label is uniformly corrected so that all modalities can give simultaneous observations of the same scene segment within the same sliding time window, laying the foundation for subsequent spatial alignment.

[0071] After timestamp alignment, perform extrinsic parameter preparation based on road geometric anchor points. The geometric anchor points are derived from stable and repeatable spatial structural features within the inspection area, including road mileage markers, tunnel portal outlines, road edge boundaries, niche structures, traffic sign corners, and intersections with road markings.

[0072] Each mode outputs the observation results of these anchor points according to its own physical imaging or ranging mechanism: the visual mode obtains the tunnel portal outline and signboard corner points through edge detection and corner point detection, and obtains the road edge and markings through line segment detection; the lidar mode obtains the tunnel portal outline and road edge lines by extracting the intersection of the strong reflection plane and the plane; the millimeter-wave radar mode can obtain a stable high-reflection echo point array at the tunnel portal and signboard; the thermal imaging mode can form temperature gradient boundaries at the edges of the tunnel portal and niche.

[0073] The observation pairs of the same physical anchor point under different modes are combined into a multimodal observation alignment sample set. Each sample contains the anchor point position under the reference mode coordinates and the anchor point position under the mode coordinates to be registered, as well as the corresponding time label, to ensure that they come from the same sliding time window. In order to ensure the availability of the samples, only anchor points that have stable trackability between adjacent frames and consistent repeated detection are selected into the sample set.

[0074] The extrinsic parameter solution employs an iterative fitting method based on a random sampling consistency model. The specific steps are as follows: First, a minimum number of anchor point pairs are randomly selected from the multimodal observation alignment sample set to fit a spatial extrinsic parameter transformation model. This model is a rigid body transformation containing rotation and translation components. Based on this candidate transformation, the remaining anchor points in the mode to be registered are projected onto the reference modal coordinate system. The coincidence deviation of each anchor point pair is calculated. Anchor point pairs with deviations less than a preset tolerance are considered interior points, and the rest are exterior points. This process of random sampling, fitting, and then statistical analysis of interior points is repeated several times. The candidate transformation with the largest number of interior points and the smallest average deviation is retained as the initial extrinsic parameter value for the current moment. Then, a full refinement is performed using the interior point set corresponding to this candidate transformation to obtain the spatial extrinsic parameter transformation matrix for the current moment.

[0075] The historical change trajectory of the extrinsic parameters is continuously recorded, and the rate of change of the extrinsic parameters is calculated. The change amplitude of the rotation component is measured by comparing the attitude difference corresponding to the previous and current extrinsic parameter matrices. That is, the maximum change angle is obtained by finding the angle change of the two attitudes in the three principal axis directions and taking it as the attitude change amplitude. The change amplitude of the translation component is measured by calculating the length difference of the translation vector in the two extrinsic parameters. The above two types of change amplitudes are divided by the time interval between the two extrinsic parameter estimates to obtain the rotation change rate and the translation change rate. The larger rate is taken as the extrinsic parameter change rate at that moment.

[0076] To suppress the cumulative error caused by equipment structural deformation and thermal drift, an external parameter change rate threshold is set. The external parameter change rate threshold is calibrated under experimental conditions based on the structural stability and thermal drift limit of the specific equipment. Under constant environment and standard attitude, the normal fluctuation range of the external parameter change rate is recorded when no external disturbance is applied. The upper limit of this range is combined with the allowable safety margin of the equipment to determine the threshold. During online operation, the external parameter update cycle parameter is used as the unit for monitoring. When the external parameter change rate of multiple consecutive external parameter update cycles exceeds the external parameter change rate threshold, the current fusion alignment state is determined to be in an abnormal drift state.

[0077] Once the abnormal drift state is confirmed, the extrinsic parameter rollback mechanism is triggered: the current extrinsic parameters are reset to the historical extrinsic parameters of the most recently marked stable state extrinsic parameter matrix, and data reprojection is immediately performed on the results that have been fused but not yet published within the current sliding time window. That is, the original observations of each mode are remapped to a unified coordinate system using the rolled-back stable extrinsic parameters. At the same time, the persistent writing of the abnormal extrinsic parameters and their corresponding fusion results is stopped. The selection rule for the stable state extrinsic parameter matrix is: when the rate of change of the extrinsic parameters has been continuously lower than the threshold for a period of time, and the average deviation of the anchor point reprojection based on the extrinsic parameter is less than the preset tolerance, the extrinsic parameter is marked as stable and cached.

[0078] Through the above implementation process, the system, under the premise of achieving timestamp alignment by adopting the cross-flow cross-correlation peak positioning method, robustly solves the cross-modal extrinsic parameters by leveraging the random sampling consistency model based on road geometric anchor points. Under the mechanism of continuously monitoring the rate of change of extrinsic parameters, it achieves real-time identification and rollback correction of abnormal drift states. Timestamp alignment ensures that multimodal observations are in the scene slices corresponding to the same moment. Geometric anchor points and random sampling consistency models ensure the anti-outlier capability of spatial registration. The extrinsic parameter change rate threshold and extrinsic parameter rollback mechanism ensure self-stability during long-term operation. Data reprojection ensures the consistency and traceability of the results after coordinate unification.

[0079] Step 3: Based on the established inspection anomaly truth table rules, and combining coarse screening features and category features, perform event validity verification, output the valid event set, and mark results that do not meet sufficient conditions or exhibit mutually exclusive triggers as review candidates. Specific steps include:

[0080] The candidate event list is obtained from the preceding sequence. Each candidate event is accompanied by a spatial grid cell number, anomaly category, trigger mode type list, trigger time interval, and mutual exclusion check results. To ensure consistent judgment criteria, parameter calibration is completed before going live, and the following parameter names are fixed: sliding time window length parameter, continuous frame number threshold parameter, pixel area threshold parameter, time overlap ratio threshold parameter, echo stability threshold parameter, temperature difference threshold parameter, diffusion continuous frame number threshold parameter, connected component length threshold parameter, contrast reduction threshold parameter, and texture decoherence threshold parameter. The coarse screening features are the foreground connected component contour, area, and continuous frame number on the visual side, the continuous echo existence and echo amplitude / distance stability on the radar side, and the temperature field connected component and temperature difference diffusion trajectory on the thermal imaging side. The category feature information is the category label output by the detection model for four types of anomalies: foreign objects, water accumulation, smoke, and missing markings, along with the corresponding morphological elements (such as closed contours, specular highlights, texture decoherence, and marking breakage direction).

[0081] The inspection anomaly truth table is used as a set of rules for cross-modal necessary and sufficient conditions. For the four types of anomalies, at least the following minimum sufficient condition combinations are set: Foreign Objects: Combination A is a visually closed contour with stable radar echo. A visually closed contour refers to a closed boundary formed within the current sliding time window, with a connected region area not less than the pixel area threshold parameter, and maintaining boundary continuity in frames with a duration threshold parameter. Stable radar echo refers to the echo distance and echo amplitude of the same spatial grid cell continuously existing within the allowable fluctuation range in the same time interval. The judgment method is to statistically analyze the difference between the maximum and minimum echo amplitude and the difference between the maximum and minimum distance within the time interval. If both are less than the echo stability threshold parameter obtained according to the equipment calibration, it is judged as stable. Combination B is visually continuous satisfaction and thermal image hot / cold anomaly. Visually continuous satisfaction means that the visual category label exists continuously in frames with a duration threshold parameter. Thermal image hot / cold anomaly means that the thermal image has a connected region within the spatial grid with a temperature difference from the background not less than the temperature difference threshold parameter, and maintains centroid stability or reasonable displacement according to the environmental wind direction in adjacent frames. Water Accumulation: Combination A consists of visual specular reflection and smoothed boundaries, along with a low-temperature connected region in the thermal image. Specular reflection and smoothed boundaries are achieved by simultaneously satisfying the contrast reduction threshold parameter and the boundary curvature continuity judgment; the low-temperature connected region in the thermal image refers to a region within the current window where the temperature is lower than the surrounding background and is continuous in the expansion direction. Combination B consists of visual specular reflection and smoothed boundaries, along with radar near-ground echo attenuation. Radar near-ground echo attenuation is determined by comparing the amplitude difference between the reference echo under dry conditions on the same road segment and the current echo to see if it exceeds the attenuation magnitude of the echo stability threshold parameter. Smoke: The unique combination consists of visual smoke texture and thermal image temperature diffusion characteristics. Visual smoke texture refers to localized contrast reduction and texture decoherence within the window, exhibiting spatiotemporal continuity in consecutive frames; the thermal image temperature diffusion characteristics are that the thermal image shows a temperature gradient expansion from the center outwards across multiple frames, with the number of consecutive frames not less than the diffusion consecutive frame threshold parameter. Marker line defects: Combination A indicates a persistent visual mark line break, with the thermal image temperature field consistent along the break point with the surrounding area. This means that a visual discontinuity is detected in the connected domain of the mark line along its length, and the break length is not less than the connected domain length threshold parameter, remaining stable for several consecutive frames. Simultaneously, no significant temperature anomalies are observed at the break point in the thermal image, ruling out water stains or foreign object obstruction. Combination B indicates a persistent visual mark line break, but no stable echo from the radar. This indicates that no stable prominent echo is observed in this area, thus ruling out misjudgment of visual breaks caused by physical obstruction.

[0082] The event validity verification process is as follows:

[0083] For each candidate event, its anomaly category and trigger mode type list are read one by one. First, it is confirmed that the event has passed the previous consistency gating and the mutual exclusion check result is recorded as not triggered. Then, within the current sliding time window, it is matched according to the minimum sufficient condition combination of the truth table: if a candidate event satisfies any minimum sufficient condition combination under its corresponding category, the candidate event is confirmed as a valid event and written into the valid event set; if a candidate event has a contradictory combination that is consistent with the mutual exclusion rejection rule during the matching process (e.g., visual smoke texture but no temperature diffusion feature in thermal image, or visual closed contour but no stable echo in radar, or thermal image cold and hot anomalies but neither visual nor radar continuously detects the object), the system marks the candidate event as a verification candidate, does not enter the valid event set, and transfers it to the secondary acquisition or manual verification queue. For example, visual continuous satisfaction is compared with the frame count to see if the continuous frame number threshold parameter is reached; radar stable echo is compared with the difference between the maximum and minimum values ​​within the sliding time window and the echo stability threshold parameter; temperature diffusion feature is judged by whether the area of ​​the temperature connected domain in multiple frames increases monotonically and whether the boundary moves continuously in adjacent frames.

[0084] Once an event is confirmed as a valid event, a structured record is generated and written into the valid event set. The structured record includes at least: the spatial grid cell number and its center coordinates in a unified coordinate system, the anomaly category, the list of triggering modal types, the triggering time interval, the minimum sufficient condition combination number used (explicitly marked as "combination A" or "combination B" in this embodiment, etc.), and the mutual exclusion check result (marked as not triggered). For events marked as review candidates, the system also generates a record but notes the review reason in the fields (e.g., hitting the mutual exclusion rule: visual smoke texture but thermal image has no temperature diffusion feature), and submits the record to the subsequent review process.

[0085] Step 4: Construct a finite state machine to manage events hierarchically. Perform state transitions based on duration, spatial coverage monotonicity, and discrete criteria for continuous, non-triggering cycles, and generate inspection records. Specific steps include:

[0086] The event state set is defined as pending state, confirmed state, escalation state, and recovery state, and a finite state machine with explicit state transition paths is established. To ensure consistent judgment criteria, the following parameter names are fixed before online operation: sliding time window length parameter and window step size parameter (for unifying time slices), effective duration threshold parameter (for transitioning from pending state to confirmed state), number of spatial coverage monotonicity judgment windows parameter (for detecting the duration of monotonically decreasing spatial coverage), and continuous non-triggering cycle threshold parameter (for falling back to the recovery state). Among them, the spatial coverage range is defined as the number of elements in the set of spatial grid cells occupied by the event within the current sliding time window; the anomaly intensity index is defined as the intensity quantification value provided by each event evidence mode (e.g., closed contour integrity and connectivity stability on the visual side, echo amplitude stability on the radar side, and temperature difference connectivity stability on the thermal imaging side).

[0087] The transition from the pending state to the confirmed state determines the effective duration. Valid events that have passed consistency gating and mutual exclusion and have been confirmed as valid are initialized to the pending state. Subsequently, within consecutive sliding time windows, valid trigger counts are performed on the event, i.e., whether the event continuously satisfies the minimum sufficient condition combination corresponding to its category within each window. The calculation process is as follows:

[0088] The system sequentially reads adjacent windows on the timeline. If an event still meets the sufficient conditions for its category within the current window, the effective duration count is incremented by one. Once a window that does not meet the sufficient conditions appears, the counting stops in this round of judgment. If the duration converted from the count value is not less than the effective duration threshold parameter (the threshold is set by the number of windows or the equivalent number of seconds, which are uniquely determined during system initialization), then the event is determined to meet the effective duration requirement and transitions from the pending state to the confirmed state; otherwise, it remains in the pending state and continues to count again in the next window sequence.

[0089] The transition from the confirmed state to the escalation state is characterized by a determination of either spatial coverage monotonicity or a continuous increase in anomaly intensity. Upon entering the confirmed state, spatial coverage and anomaly intensity indices are monitored independently. The determination of spatial coverage monotonicity is as follows: within a continuous window sequence (not less than the number of windows specified by the spatial coverage monotonicity determination parameter), the number of spatial grid cells occupied by the event is calculated window by window, and the coverage quantity of adjacent windows is compared one by one. If no decrease is observed in each comparison (equality is allowed; no increase is not considered a decrease), then the spatial coverage is considered monotonic and undiminished in that sequence. The determination of a continuous increase in anomaly intensity is as follows: within a continuous window sequence of the same length, the intensity quantization values ​​of at least two different modalities do not decrease between adjacent windows, and there is at least one increase in the entire sequence. This indicates a continuous increase in the anomaly intensity index. When either of these two criteria is met, the transition from the confirmed state to the escalation state is initiated to reflect an increase in event priority or urgency of handling. If neither criterion is met, the observation continues in the confirmed state.

[0090] The transition from the confirmation state and the escalation state to the recovery state is characterized by continuous no-trigger cycle judgment and fallback. When an event is in the confirmation state or the escalation state, the system performs a no-trigger detection once in each newly arrived sliding time window. The no-trigger judgment is: within the current window, the event no longer satisfies any minimum sufficient condition combination of its category in the inspection anomaly truth table; if there is only a single modality sporadic triggering but no sufficient condition is formed, it is also counted as no triggering. Once all events are judged as no triggering within the number of windows specified by the continuous no-trigger cycle threshold parameter, the current state is transitioned to the recovery state. After transitioning to the recovery state, the system marks the event as pending clearing or completed follow-up, which means that low-priority review can be arranged on-site or the work order can be closed directly.

[0091] At each state change, the system synchronously generates a structured inspection record. The record items include at least: a unique event identifier, a state name (pending state / confirmed state / escalated state / recovery state), entry time and exit time, a state transition path (e.g., from pending state to confirmed state to escalated state to recovery state), criteria for triggering the transition (e.g., the effective duration reaches the effective duration threshold parameter, spatial coverage monotonicity is met, and the continuous no-trigger cycle reaches the continuous no-trigger cycle threshold parameter), the corresponding window index range, and a summary of key information used for the judgment, such as whether the sequence of spatial grid cell numbers involved and the window sequence of modal intensity quantization values ​​have a non-decreasing relationship.

[0092] Step 5: Based on the constructed topology map, perform multi-platform collaborative scheduling, select the reachable scheduling path with the shortest path length, and conduct on-board or airborne platform verification and re-inspection, while coordinating with on-site facilities to complete the response. Specific steps include:

[0093] A topology map is established based on road structure data, monitoring equipment deployment data, and platform dwell point information for the inspection area. The topology map includes nodes and edges: nodes represent the spatial locations of vehicle-mounted platform dwell points, airborne platform take-off and landing points, and on-site facilities (including information boards, lighting, fans, broadcasting devices, etc.); edges represent the traffic accessibility paths between nodes and their distance attributes. The distance attributes are obtained from the road segment centerline length or flight route planning length in the high-precision map and are uniformly labeled in the same coordinate system. To ensure the accuracy and effectiveness of accessibility, edges also record restriction information, including road traffic restrictions (such as height restrictions, width restrictions, and closures), flight restrictions (such as no-fly zones and permitted flight ranges inside tunnels), and construction control information. When the control status changes, the system immediately updates the affected edges to an unavailable state. The above nodes and edges constitute a static skeleton, while the platform online status (battery power, load, communication link) and weather elements (wind speed, rainfall) serve as dynamic supplementary information for the topology map, used to correct accessibility judgments at any time.

[0094] Once an event enters the confirmed or escalated state, the scheduling process is initiated. First, a candidate platform set is generated: Platforms with usable status are selected from vehicle-mounted and airborne platforms. A platform is considered usable if it simultaneously meets the following three conditions: battery level is not lower than the battery threshold parameter; the task payload meets the sensor configuration required for evidence collection or re-inspection and the sensor health indicators are normal; and the communication link is available. Subsequently, on the topology graph, using the target node corresponding to the event location as the endpoint, the shortest path is calculated for the starting node of each candidate platform: the system progressively accumulates distance attributes across all available edges, selecting the reachable path with the smallest total distance that does not trigger any restrictions, and records the predecessor relationships on the path to generate a road sign sequence (for vehicle-mounted platforms) or a waypoint sequence (for...). For airborne platforms, if multiple reachable paths exist with the same total distance, the estimated arrival time is compared: the estimated arrival time is calculated segment by segment from the platform type's average travel speed parameter and the path length, and the platform with the shorter estimated arrival time is selected; if the estimated arrival times are still the same, the platform with the shorter task queue length is selected to reduce queuing. After determining the reachable scheduling path with the shortest path length and the corresponding platform, the system issues scheduling instructions: the vehicle-mounted platform travels according to the road sign sequence to the vicinity of the event grid to perform close-range evidence collection or re-inspection, and the airborne platform takes off from the compliant take-off and landing point and flies along the waypoint sequence to the designated position above the target or outside the tunnel entrance to perform aerial evidence collection or re-inspection; if tunnels or no-fly zones are involved, the airborne platform's waypoints are only planned up to the permitted boundary, and the remaining close-range re-inspection is completed by the vehicle-mounted platform.

[0095] Simultaneously with the platform's deployment, on-site facilities are coordinated to complete the response. The response is based on a response template, which is set according to the event category and status: when the event is a confirmed "foreign object" or "water accumulation," speed limit warnings and lane change guidance are sent to the information boards of nearby nodes; when the event is escalating, lane closures and warning broadcasts are added; for events such as smoke or sudden drops in visibility, enhanced lighting and ventilation fan start / stop strategies are coordinated within the tunnel. The coordination command is sent in the form of node number and action parameters, and on-site facilities return execution receipts and timestamps; if the specified feedback time limit is met... If no valid response is received, the system marks the facility as a failed linkage and selects an adjacent facility node to issue an equivalent action. At the same time, the anomaly is written to the scheduling log. After the platform completes the evidence collection or re-inspection, the arrival time, path number, evidence material identifier, and on-site facility response are summarized and written to the event record for closed-loop traceability and strategy optimization. The entire process does not rely on weighted summation or scoring models. All choices are based on the minimum path length, platform availability, and non-triggered constraints as clear criteria, ensuring that path selection is objective and transparent, scheduling decisions are reviewable, and handling linkage is traceable.

[0096] Step 6: Perform path calibration and event location alignment in areas with limited positioning signals, and update the inspection strategy and rule entries based on historical inspection data to achieve dynamic optimization. Specific steps include:

[0097] When entering areas with limited positioning signals, such as tunnels, under bridges, or in canyons, the path calibration mechanism is first activated. Based on the mileage reference line information (indexed by station number and arc length along the route) pre-collected and stored in the road segment base map, as well as the spatial coordinates of fixed geometric anchor points on walls and structures (including tunnel portal outline inflection points, niche edges, signboard corner points, curb line inflection points, etc.), a path fingerprint sequence is constructed for that road segment. The path fingerprint sequence consists of a four-tuple sequence ordered by mileage: anchor point identifier, then mileage station number, then arc length along the route, and finally anchor point type. The spatial coordinates of each anchor point correspond one-to-one with its projected arc length position on the mileage reference line, which is used for subsequent sequence matching. Within a confined area, the inspection platform continuously outputs inertial measurement unit data (time series of acceleration and angular velocity) and a sequence of geometric anchor points observed along the route (anchor point identifiers and observation times identified by cameras and radar under a unified time reference). Within a fixed path calibration window length parameter, it performs sliding matching on the relative start and end positions of the online anchor point sequence and the path fingerprint sequence. The optimal alignment segment is determined based on the criteria that the matched anchor point order is consistent and the cumulative deviation of the arc length difference between adjacent anchor points does not exceed the anchor point matching tolerance parameter. When the above matching criteria are met within a continuous window and the cumulative deviation of the arc length difference at both ends of the matched segment is within the tolerance, loop closure correction is triggered (the loop closure trigger threshold is jointly given by the minimum number of matched anchor points threshold parameter and the cumulative deviation tolerance parameter).

[0098] After matching is completed, error minimization fitting and positioning trajectory correction are performed on the platform positioning trajectory: Within the matching segment, the difference between the projected arc length position of the platform trajectory point on the mileage reference line and the arc length position of the anchor point in the path fingerprint sequence is calculated for each anchor point, and these differences are arranged into an error sequence in chronological order; then, taking the temporally continuous and consistent segments of the error sequence as units, the average difference of the segment is smoothly distributed as the arc length offset correction amount to the trajectory arc length position at each sampling time within the segment, thereby obtaining the drift-free arc length coordinates; in the lateral direction, the lateral correction amount is limited to not exceeding the horizontal direction using the normal vector of the matching anchor point as a reference. To correct the upper limit parameter and avoid excessive oscillation, after the trajectory correction is completed, the position of the abnormal target is described using a standardized coordinate system (i.e., the system unified coordinate system): first, the arc length position of the corrected trajectory at the time of the anomaly is taken as the reference point along the line, and then the measured lateral offset is used as the lateral reference quantity, which is written as a spatial reference expression of (arc length position, lateral offset, lane side). If the abnormal target is given by multiple modes, the event position is aligned once based on the corrected trajectory at the same time under the unified clock. The same spatial reference point and time label are used for different modal observations of the same abnormal event to ensure consistent positioning across modes and multiple platforms.

[0099] Periodic statistical analysis is performed on historical inspection data. The statistical objects include: the frequency distribution of triggering of each anomaly category in different road sections and time periods; the conversion ratio from valid candidate events to confirmed and escalated states; the proportion of mutually exclusive triggers marked as review candidates; the success rate of trajectory correction and the consistency rate of event location alignment in restricted areas; and the original records corresponding to typical interference scenarios (such as visual smoke misjudgment caused by strong backlighting and instantaneous thermal image anomalies caused by alternating hot and cold temperatures at tunnel entrances) are extracted to locate the cause of strategy mismatch.

[0100] Based on the above statistical results, the inspection strategy parameters were adjusted (including sliding time window length, time overlap ratio threshold, continuous frame count threshold, echo stability threshold, temperature difference threshold, diffusion continuous frame count threshold, connected component length threshold, path calibration window length, anchor point matching tolerance, and lateral correction upper limit). The rule entries in the anomaly category truth table were updated: for combinations of minimum sufficient conditions with low confidence, entries were added or deleted, or threshold ranges were adjusted; for categories with multiple combinations, priority was given based on historical validity performance. All candidate changes were first verified in offline playback of historical data to ensure both improved accuracy and stability (accuracy can be measured by the improved ratio of valid events to review candidates, and stability can be measured by the absence of an increase in state transition jitter). After successful verification, new rule entry versions and strategy parameter versions were generated, synchronized to the task management module and edge running nodes, and the previous rollback version was retained. After deployment, continuous monitoring was conducted; if an abnormal increase occurred, automatic rollback to the previous stable version was implemented.

[0101] Through the above implementation, path calibration and event location alignment can be achieved in areas with limited positioning signals, and long-term robustness and adaptability can be maintained through history-driven strategy and rule updates.

[0102] It should be noted that the thresholds involved in the embodiments can be determined according to specific scenarios and needs.

[0103] This invention deploys camera equipment, radar devices, thermal imaging terminals, and vehicle-mounted and airborne platforms in highway inspection scenarios to collect multi-source sensing data. Within a unified time window, Boolean consistency gating and mutual exclusion rules are used to cross-validate the detection results of different modalities, eliminating logically contradictory triggering events and generating a highly reliable candidate event set, effectively improving the accuracy and reliability of anomaly identification. Furthermore, a stochastic consistency model is constructed based on road geometric anchor points to dynamically solve cross-modal extrinsic parameters. Combined with a rollback mechanism triggered by a threshold for the rate of change of extrinsic parameters, alignment distortion caused by equipment drift and thermal noise is avoided, ensuring the uniformity and long-term stability of multimodal data in spatial coordinates.

[0104] By combining the truth table rules of inspection anomalies, necessary and sufficient conditions are set for various anomalies such as foreign objects, water accumulation, smoke, and missing road markings, enabling automatic confirmation and hierarchical management of events. The life cycle evolution of events is tracked through a finite state machine, accurately reflecting the occurrence, escalation, and resolution of events. At key nodes of confirmation or escalation, multi-platform collaborative scheduling is performed based on the topology map, intelligently selecting the scheduling path with the shortest path length and reachability. The vehicle-mounted and airborne platforms are coordinated to efficiently complete evidence collection or re-inspection, and the on-site facilities are linked to trigger the response. This achieves closed-loop management of the entire chain from discovery to handling, thereby significantly improving inspection efficiency and reliability, reducing labor costs and misjudgment rate, and possessing the technical advantages of intelligence, globalization, and long-term self-adaptation.

[0105] Example 2: A highway three-dimensional intelligent inspection system based on multi-source sensing, such as Figure 2 As shown, it specifically includes:

[0106] The candidate event generation module is used to generate candidate events based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios. Within the synchronization time window, it performs Boolean consistency gating on multimodal outputs falling into the same grid and applies mutual exclusion rules.

[0107] The event analysis module performs spatiotemporal alignment and fusion processing on candidate events, uses time cross-correlation for timestamp alignment, solves cross-modal extrinsic parameters based on the stochastic consistency model of road geometric anchor points, and sets a threshold for the rate of change of extrinsic parameters to trigger extrinsic parameter rollback, thus completing coordinate unification.

[0108] The anomaly review module performs event validity verification based on the set inspection anomaly truth table rules and combines coarse screening features and category features. It outputs a set of valid events and marks results that do not meet sufficient conditions or are mutually exclusive as review candidates.

[0109] The inspection hierarchical management module constructs a finite state machine to manage events hierarchically. It performs state transitions based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and generates inspection records.

[0110] The dispatch and handling module performs multi-platform collaborative dispatch based on the constructed topology map, selects the reachable dispatch path with the shortest path length, and conducts on-board or airborne platform verification and re-inspection, and coordinates with on-site facilities to complete the handling response;

[0111] The inspection strategy update module performs path calibration and event location alignment in areas with limited positioning signals, and updates the inspection strategy and rule entries based on historical inspection data to achieve dynamic optimization.

[0112] The above formulas are all dimensionless calculations. Dimensionless calculations can be performed using various methods such as standardization, which will not be elaborated here. The formulas are derived from software simulations based on a large amount of collected data, and the preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0113] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, ATA hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state ATA hard disk.

[0114] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0115] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0118] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent three-dimensional highway inspection based on multi-source sensing, characterized in that, Includes the following steps: Based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios, Boolean consistency gating is performed on multimodal outputs falling into the same grid within the synchronization time window, and mutual exclusion rules are applied to generate candidate events. Candidate events are spatiotemporally aligned and fused. Time cross-correlation is used for timestamp alignment. Cross-modal extrinsic parameters are solved based on the stochastic consistency model of road geometric anchor points. An extrinsic parameter change rate threshold is set to trigger extrinsic parameter rollback and complete coordinate unification. Based on the established inspection anomaly truth table rules, the validity of events is confirmed by combining coarse screening features and category features. A set of valid events is output, and results that do not meet the sufficient conditions or are mutually exclusive are marked as review candidates. A finite state machine is constructed to manage events hierarchically. State transitions are performed based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and inspection records are generated. Based on the constructed topology map, multi-platform collaborative scheduling is performed. The reachable scheduling path with the shortest path length is selected, and the vehicle-mounted or airborne platform is used for verification and re-inspection. The on-site facilities are also linked to complete the handling and response. In areas with limited positioning signals, path calibration and event location alignment are performed, and inspection strategies and rule entries are updated in conjunction with historical inspection data to achieve dynamic optimization; Within the synchronization time window, Boolean consistency gating is performed on multimodal outputs falling into the same grid, and mutual exclusion rules are applied to generate candidate events. The specific process is as follows: Under a unified time reference, a sliding time window is set to perform time-series alignment of abnormal target information collected by different types of sensing terminals, and grid areas are divided based on the road spatial structure to map multimodal sensing results with spatiotemporal labels to the corresponding spatial grid units. When the same grid cell receives anomaly category labels from different modalities within the same time window, it is judged according to the set Boolean consistency gating condition. If at least two different modal devices trigger detection results for the same anomaly category within the time window, and the time overlap ratio meets the set threshold, the anomaly is determined to be a consistent trigger and enters the candidate event list. Under the premise that Boolean consistency is true, a mutual exclusion rule is applied for screening. The mutual exclusion rule is a combination relationship in which there is a physical or logical contradiction between different modalities. When mutually exclusive combinations exist, the event triggering result is either rejected or downgraded and marked as a review candidate. Events that satisfy Boolean consistency gating and are not mutually exclusive are output as valid candidate events, and their corresponding grid positions, anomaly categories, trigger mode types, and time intervals are recorded.

2. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 1, characterized in that: The mutually exclusive rejection rules include visual recognition of a closed contour but no radar echo, visual detection of smoke texture but no temperature diffusion characteristics in thermal imaging, and thermal imaging showing cold and hot anomalies but neither visual nor radar continuously detects the object.

3. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 1, characterized in that: Using time cross-correlation for timestamp alignment refers to using the cross-current cross-correlation peak positioning method for alignment. The threshold for the rate of change of external parameters is set according to the equipment structure and the upper limit of thermal drift. When the continuous detection exceeds the threshold, external parameter rollback and data reprojection are performed.

4. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 3, characterized in that: The stochastic consistency model based on road geometric anchor points is used to solve cross-modal extrinsic parameters, and a threshold for the rate of change of extrinsic parameters is set to trigger extrinsic parameter rollback, thereby completing coordinate unification. The specific process is as follows: The spatial structural features deployed in the inspection area are used as geometric anchors, including road mileage markers, tunnel portal outlines, road edge boundaries, niche structures, traffic sign corners and intersections of road markings; Based on the observation results of geometric anchor points in different modes, a multimodal observation aligned sample set is constructed. The random sampling consistency model is used to iteratively fit the sample set and solve the spatial extrinsic transformation matrix between each mode at the current time. During continuous operation, the historical change trajectory of the external parameters is recorded, and the rate of change of the external parameters is calculated. The rate of change of the external parameters is the change amplitude of the rotation and translation components in the external parameter matrix per unit time. When the rate of change of the extrinsic parameters exceeds the set threshold and the parameters are in an abnormal drift state for multiple consecutive time periods, the current fusion alignment state is determined to be unreliable, triggering the extrinsic parameter rollback mechanism. The current extrinsic parameters are reset to the extrinsic parameter matrix that was previously marked as stable, and the writing of the fusion results for this round is stopped.

5. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 4, characterized in that: The inspection anomaly truth table specifies the minimum sufficient condition combination for foreign objects, water accumulation, smoke and road marking defects. Among them, foreign objects are confirmed to meet any combination of visual closed contour and stable radar echo or visual continuous satisfaction and thermal image cold and hot anomalies. Mutually exclusive candidate events are automatically downgraded to review candidates.

6. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 5, characterized in that: A finite state machine is constructed to manage events hierarchically. State transitions are executed based on duration, spatial coverage monotonicity, and discrete criteria with continuous, non-triggering cycles, and inspection records are generated. The specific process is as follows: Define the event state set as including undetermined state, confirmed state, escalation state and recovery state, and construct a finite state machine with explicit state transition paths to describe the life cycle evolution process of the event; Events in the pending state are connected to a finite state machine. When the event satisfies the consistency triggering condition and the effective duration within the continuous observation window is greater than the preset duration threshold, it transitions from the pending state to the confirmed state. After an event enters the confirmed state, if the spatial coverage of the abnormal target shows a monotonically increasing trend in a continuous time window, or if the abnormality intensity index continues to increase, the transition from the confirmed state to the escalation state is triggered. If, after an event enters the escalation or confirmation state, no abnormal target is detected again within a set number of consecutive time windows, the event transitions from the current state to the recovery state and is marked as pending clearance or completed revisit. During the event state change process, the entry time, exit time, state transition path, and criteria for triggering the transition for each state are recorded, and structured inspection records are generated simultaneously.

7. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 6, characterized in that: Based on the constructed topology map, multi-platform collaborative scheduling is performed. The reachable scheduling path with the shortest path length is selected, and the vehicle-mounted or airborne platform conducts verification and re-inspection, and coordinates with on-site facilities to complete the handling response. The specific process is as follows: Based on the road structure, monitoring equipment deployment location and platform dwell point information within the inspection area, a topology graph is constructed. The topology graph contains nodes and edges. Nodes represent the spatial location of vehicle-mounted platforms, airborne platforms or field equipment, and edges represent the traffic reachable paths between nodes and their distance or cost attributes. Once an event enters the confirmed or escalated state, based on a comprehensive assessment of the event location and platform reachability, the shortest path is calculated using a topology graph, prioritizing the scheduling path with the shortest path length and available platform status. Depending on the platform type, vehicle-mounted or airborne platforms may be dispatched to perform certification or re-inspection tasks. Simultaneously, the on-site facilities deployed near the affected area are coordinated, and response actions are controlled and issued according to the event type and the corresponding handling template.

8. The method for intelligent three-dimensional highway inspection based on multi-source sensing according to claim 7, characterized in that: In areas with limited positioning signals, path calibration and event location alignment are performed, and inspection strategies and rule entries are updated based on historical inspection data to achieve dynamic optimization. The specific process is as follows: In areas with limited signal coverage, a path calibration mechanism is activated to construct a path fingerprint sequence based on mileage reference line information deployed on the target road segment and fixed geometric anchor points on walls and structures. The collected inertial measurement unit data and the geometric anchor point sequence observed along the route are used to perform path matching and error minimization fitting to complete the loop correction and positioning trajectory correction of the platform's running path, and to uniformly map abnormal targets to spatial reference points in the standardized coordinate system. By combining historical inspection data recorded by the system, statistical analysis is performed on different anomaly categories to identify interference scenarios. Adjust the inspection strategy parameters, and add, delete, adjust threshold ranges, or sort priorities of the necessary and sufficient condition entries in the anomaly category truth table to form a new version of the rule entries; The updated inspection strategy and rule entries are synchronized to achieve dynamic optimization of inspection tasks.

9. A highway three-dimensional intelligent inspection system based on multi-source perception, used to implement the highway three-dimensional intelligent inspection method based on multi-source perception as described in any one of claims 1-8, characterized in that, include: The candidate event generation module is used to generate candidate events based on multimodal perception data collected by camera equipment, radar devices, thermal imaging terminals, vehicle-mounted and airborne platforms deployed in highway scenarios. Within the synchronization time window, it performs Boolean consistency gating on multimodal outputs falling into the same grid and applies mutual exclusion rules. The event analysis module performs spatiotemporal alignment and fusion processing on candidate events, uses time cross-correlation for timestamp alignment, solves cross-modal extrinsic parameters based on the stochastic consistency model of road geometric anchor points, and sets a threshold for the rate of change of extrinsic parameters to trigger extrinsic parameter rollback, thus completing coordinate unification. The anomaly review module performs event validity verification based on the set inspection anomaly truth table rules and combines coarse screening features and category features. It outputs a set of valid events and marks results that do not meet sufficient conditions or are mutually exclusive as review candidates. The inspection hierarchical management module constructs a finite state machine to manage events hierarchically. It performs state transitions based on discrete criteria such as duration, spatial coverage monotonicity, and continuous non-triggering cycles, and generates inspection records. The dispatch and handling module performs multi-platform collaborative dispatch based on the constructed topology map, selects the reachable dispatch path with the shortest path length, and conducts on-board or airborne platform verification and re-inspection, and coordinates with on-site facilities to complete the handling response; The inspection strategy update module performs path calibration and event location alignment in areas with limited positioning signals, and updates the inspection strategy and rule entries based on historical inspection data to achieve dynamic optimization.

Images