Unmanned aerial vehicle-based expressway thrown object automatic identification system

Abstract

The invention, which relates to the technical field of intelligent traffic and unmanned aerial vehicles, discloses an unmanned aerial vehicle-based automatic identification system for a thrown object on a highway, comprising: an unmanned aerial vehicle cluster deployment module which adopts a Mesh ad hoc network and covers a lane with a width of 150-250 m; the multi-modal data acquisition module is used for acquiring infrared light, visible light, a depth map and point cloud data; the image preprocessing module is used for downsampling, filtering and denoising and enhancing the contrast ratio; the thrown object recognition module is fused with multi-modal features, the confidence coefficient threshold value is 0.6-0.7, and suspected targets are verified through point cloud clustering; the three-dimensional modeling and positioning module is used for fusing binocular and laser radar data; the intelligent decision-making and scheduling module is used for generating priorities in combination with the traffic flow and planning an inspection path; and the linkage processing module is used for pushing information through 4G / 5G, and linking the information board and the road administration vehicle for processing. The method improves the monitoring coverage rate and the recognition accuracy, shortens the response time, reduces the positioning error, optimizes the processing efficiency, and guarantees the high-speed traffic safety.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and unmanned aerial vehicle (UAV) technology, and in particular to an automatic identification system for highway spills based on unmanned aerial vehicles (UAVs). Background Technology

[0002] Highway debris, a hidden killer of road traffic safety, poses a significant increase in danger as traffic density rises. Traditional debris monitoring relies primarily on manual inspections, typically every 2-4 hours, covering an area of ​​no more than 50 kilometers per inspection. This is not only costly in terms of manpower but also inadequate for responding to sudden debris events. For example, tire fragments or cargo packaging that fall from trucks, if not detected in time, can easily cause rear-end collisions, especially at night or in adverse weather conditions, where the incidence of such accidents is 3-4 times higher than normal.

[0003] While existing fixed monitoring equipment enables real-time monitoring of some road sections, it has many limitations. The spacing between monitoring poles is typically 1-2 kilometers, easily creating blind spots in complex terrain areas such as curves and slopes. The fixed focal length of the cameras results in insufficient ability to identify small, long-distance projectiles (such as metal pieces less than 30 centimeters in diameter), leading to a false negative rate exceeding 40%. Some pilot projects using single-drone inspection solutions are limited by battery life (usually only 30-40 minutes) and data processing capabilities, making it difficult to cover long distances. Furthermore, in severe weather conditions such as heavy rain and fog, the identification system relying solely on visible light cameras becomes completely ineffective.

[0004] More critically, the existing system lacks a fully intelligent handling mechanism. The accuracy of spilled material location is mostly within 5-10 meters, and road administration vehicles still need 10-15 minutes to search for the target after arriving at the scene. Prioritization is based solely on the size of the spilled material, without considering its material (such as the difference in hazard between metal and plastic), the lane it is in (the risk difference between the overtaking lane and the emergency lane), and real-time traffic flow, leading to an imbalance in resource allocation—delays in emergency response while excessive manpower is required for low-risk events. With the total length of highways exceeding 170,000 kilometers, traditional monitoring and handling models can no longer meet the safety management needs of timely detection and rapid removal. Building an intelligent, integrated spilled material identification and handling system has become an inevitable trend. Summary of the Invention

[0005] This invention proposes an automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) includes the following modules:

[0008] The drone swarm deployment module consists of one mother drone and 3 to 5 drones. It adopts a Mesh self-organizing network. The drones are distributed in a diamond array, covering a lane with a width of 150 to 250 meters. The mother drone cruises at an altitude of 120 to 180 meters, and the drones fly at an altitude of 80 to 120 meters. All drones are equipped with RTK-GPS.

[0009] Multimodal data acquisition module: Deploys a thermal imaging camera to acquire infrared images, a visible light camera to capture color video, a binocular camera to generate depth maps, and a 64-256 line lidar to acquire 3D point clouds. All devices achieve time synchronization through PPS synchronization pulses.

[0010] Image preprocessing module: High-resolution images are downsampled to 1080p~2K using bilinear interpolation, noise is removed by 3×3~5×5 Gaussian filtering, contrast is enhanced by CLAHE algorithm, and dynamic correction is enabled for motion-blurred images.

[0011] The projectile identification module is based on the improved YOLOv8-nano to YOLOv8-s network, which integrates RGB image and depth map features. The training set contains 80,000 to 120,000 labeled samples, and the confidence threshold is set to 0.6 to 0.7. Suspected targets are verified by point cloud clustering.

[0012] 3D modeling and positioning module: The initial 3D coordinates are calculated using the binocular parallax principle. The ICP algorithm is used to fuse the point cloud with the LiDAR to construct a 3D model of the projectile. The positioning result is output after coordinate transformation, with a timestamp attached.

[0013] Intelligent decision-making and scheduling module: It obtains real-time traffic flow from the highway management platform, generates disposal priorities by combining spilled material parameters, plans inspection paths using an improved ant colony algorithm, and triggers a response mechanism within 2 to 5 minutes for emergencies;

[0014] Joint response module: Pushes alarm information through 4G / 5G private network, simultaneously controls variable message signs within 1-3km, sends navigation coordinates to the nearest road administration vehicle, and records the entire response process.

[0015] Furthermore, it also includes a motion blur correction algorithm in the image preprocessing module: Where I(x,y) represents the pixel value of the blurred observed image, O(x,y) represents the pixel value of the original clear image, v represents the UAV's relative flight speed to the ground, T represents the camera exposure time, t represents the time variable, and α represents the light intensity attenuation coefficient. In practice, the motion vectors of adjacent frames are first calculated using the Lucas-Kanade optical flow method. For linear blurring caused by high-speed flight, Wiener filtering is used for deconvolution, and gradient enhancement algorithm is used simultaneously to strengthen edge features. After correction, the image quality is evaluated. When the evaluation value is improved by ≥30% to 50%, the correction is considered effective; otherwise, a backup image is used for fusion and supplementation.

[0016] Furthermore, it also includes the depth confidence calculation in the spill identification module: Among them, C d For depth confidence, σ d The standard deviation of the depth at the current measurement point. Let d be the standard deviation of the road surface background depth, d be the measured depth value at the current point, and d0 be the mean of the road surface background depth. During implementation, C is calculated for each pixel in the image. d When C d When the value is <0.2 to 0.4, it is initially identified as a foreground target. Then, the road surface interference is filtered out by combining RGB features. For areas identified as spilled objects, the minimum bounding rectangle is extracted to eliminate small-sized noise.

[0017] Furthermore, the synchronization mechanism of the multimodal data acquisition module is as follows: time synchronization is achieved using the IEEE 1588PTP protocol, with the master clock being the mother GPS module and the slave clocks being the slave devices and each sensor. The synchronization formula is: t sync =t GPS +Δt prop +Δt proc Among them, t sync t is the final timestamp of the sensor data. GPS The original GPS time, Δt prop For signal propagation delay, Δt proc To compensate for the internal processing delay of the sensor, during implementation, a synchronization calibration is performed every 5 to 15 ms. Clock drift is compensated by a phase-locked loop, and timestamp interpolation is used to align the lidar point cloud and the visible light image. In case of synchronization failure, the system automatically switches to the internal crystal oscillator of the host machine to maintain synchronization until the signal is restored.

[0018] Furthermore, the intelligent decision-making and scheduling module also includes a dynamic priority adjustment subunit: based on the real-time status changes of the spilled material, the priority is reassessed every 30 to 60 seconds, and a "risk diffusion index" is introduced; for emergency targets, drone tracking and shooting are automatically triggered, and dynamic selection annotations are added to the video stream that is simultaneously pushed to the management center; when planning the route, a "zonal disposal strategy" is adopted, dividing the highway into disposal units of 5 to 10 km, and prioritizing the dispatch of the nearest 2 to 3 road administration vehicles in each unit to avoid efficiency loss in cross-regional dispatch.

[0019] Furthermore, the drone swarm deployment module also includes an adaptive load balancing subunit: it monitors the battery level and data transmission link quality of each drone in real time, and automatically allocates the monitoring area it is responsible for to neighboring drones when the load of a drone is too high; the host drone periodically generates a swarm health report, plans return routes for drones that need to be charged, and simultaneously dispatches backup drones to fill monitoring blind spots.

[0020] Furthermore, the linkage response module also includes a multi-level early warning linkage sub-unit: Level 1 early warnings are only displayed and recorded in the management center; Level 2 early warnings simultaneously trigger variable message signs within 1-3km; Level 3 early warnings link with the highway traffic police system, pushing road closure suggestions, and simultaneously controlling drones to hover 100-150m above the spilled material, turning on flashing lights to warn passing vehicles; all early warning information is accompanied by a unique code, and the associated response results form a closed-loop record.

[0021] Furthermore, the spill identification module also includes a few-shot learning enhancement subunit: for rare spill types, transfer learning combined with data augmentation techniques is used to expand the sample to 2000-3000 images; a contrastive learning mechanism is introduced to construct "spill-background" positive and negative sample pairs, and discriminative features are learned through Siamese networks; for newly emerging unknown types, they are automatically marked as "to be classified" and pushed to manual review, and the review results are incorporated into the training set to achieve incremental model updates.

[0022] Furthermore, the 3D modeling and positioning module also includes an environmental adaptability compensation subunit: for rainy and snowy weather, noise filtering is performed on the lidar point cloud, and invalid values ​​are eliminated by dynamic thresholding of the binocular depth map; during positioning, INS inertial navigation data is fused, and position drift is corrected by Kalman filtering; after the 3D model is generated, the "dangerous volume" and "lane encroachment ratio" of the spilled material are automatically calculated to provide a quantitative basis for disposal.

[0023] Furthermore, the image preprocessing module also includes an illumination adaptive adjustment subunit: it uses HDR synthesis technology to process backlight scenes, separates the illumination component and reflection component of the image through the Retinex algorithm, and dynamically adjusts the Gamma value; for nighttime scenes, it enables the fusion of thermal imaging and visible light images to highlight the infrared characteristics of high-temperature debris, while suppressing the strong light from vehicle headlights to ensure all-weather recognition stability.

[0024] Compared with existing technologies, the beneficial effects of this invention are:

[0025] In terms of monitoring coverage, the collaborative operation of drone swarms breaks through the range limitations of a single device. Adaptive formation adjustment can dynamically optimize monitoring resources according to the density of the spilled material. Combined with a wide coverage range of 150-250 meters, it achieves blind-spot-free monitoring of the entire road section. In particular, it solves the monitoring problems of special road sections such as curves and bridges, leaving no place for various spilled materials to hide.

[0026] In terms of recognition accuracy, multimodal data fusion and intelligent preprocessing technology significantly reduce environmental interference. The complementarity of visible light, infrared and lidar data, combined with algorithms such as adaptive lighting adjustment and motion blur correction, effectively overcomes recognition obstacles in complex scenarios such as backlight, heavy rain, and nighttime, ensuring stable recognition performance in all weather conditions, reducing missed detections and false judgments, and enabling the accurate capture of small, dark-colored and easily overlooked spilled objects.

[0027] The improvement in handling efficiency is particularly significant. High-precision three-dimensional positioning and hazard quantification analysis provide road administration vehicles with centimeter-level target guidance, greatly shortening on-site search time; dynamic priority adjustment and zoned scheduling strategies, combined with a multi-level early warning and linkage mechanism, achieve optimal resource allocation, ensure rapid response to emergencies, and reduce the risk of accidents caused by spilled materials remaining on the road.

[0028] Furthermore, the system's self-learning and iterative capabilities ensure long-term applicability, while few-shot learning technology enables rapid identification of new types of spilled materials, and the incremental model update mechanism allows for continuous optimization of system performance. Overall, this system constructs a closed-loop solution from monitoring and identification to handling, providing more proactive and intelligent protection for highway traffic safety and effectively improving road safety and smoothness. Attached Figure Description

[0029] Figure 1 This is a schematic block diagram of the automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) proposed in this invention.

[0030] Figure 2 A schematic diagram showing the coverage area of ​​the drone swarm monitoring.

[0031] Figure 3 Line graph comparing the accuracy of this system and traditional systems in identifying spilled objects under different environments;

[0032] Figure 4 Radar chart for prioritizing the disposal of spilled materials. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0035] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0036] Reference Figures 1 to 4 An automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) includes the following modules:

[0037] The UAV swarm deployment module consists of one mother drone and 3-5 drones, employing a mesh self-organizing network. The drones are distributed in a diamond array, covering a lane width of 150-250m. The mother drone cruises at an altitude of 120-180m, and the drones fly at an altitude of 80-120m. All drones are equipped with RTK-GPS. Before system deployment, careful equipment calibration and parameter configuration are required. The UAV swarm, consisting of one mother drone and four drones, requires GPS differential positioning calibration, specifically through 30 minutes of static observation to ensure a positioning error of no more than 3cm. The baseline distance of the binocular cameras is set to 15cm, allowing for an error range of ±2cm. The 128-line lidar has its scanning frequency initialized to 20Hz.

[0038] The management center server needs to be configured with a MySQL 8.0 database and pre-set a spill classification library. This library covers 15 categories, including hard metal objects and plastic containers, and assigns a hazard factor of 1-3 to each type of spill. Simultaneously, a 1:1000 scale electronic map of the highway needs to be entered, including lane markings, bridges, curves, and other feature markers.

[0039] During the initialization phase, the master and slave units pair up via a mesh self-organizing network operating in the 5.8GHz communication band with a transmission rate of 100Mbps. The slave units are arranged in a diamond array, with a horizontal spacing of 50 meters and a vertical spacing of 30 meters, forming a monitoring strip with a width of 200 meters. The master unit's cruising altitude is set to 150 meters, and the slave units' cruising altitude is set to 100 meters. The flight speed is uniformly set to 80 km / h, and the flight direction is consistent with the direction of traffic flow. Figure 2 The diagram illustrates the monitoring coverage of a drone swarm, visually showcasing a diamond-shaped array layout consisting of one mother drone and four sub-drones. The mother drone is located at the center, with the sub-drones spaced 50 meters apart laterally and 30 meters apart longitudinally. The mother drone covers the entire lane at a cruising altitude of 150 meters, while the sub-drones each cover a portion of the lane at an altitude of 100 meters, forming a 200-meter-wide, non-overlapping monitoring zone. This clearly demonstrates the monitoring range and collaborative coverage effect of the drone swarm under the Mesh self-organizing network.

[0040] The drone swarm deployment module includes an adaptive load balancing subunit. This subunit monitors the remaining battery level of each drone in real time and issues an alert when the remaining battery level falls within the 20%-30% threshold range. It also monitors the data transmission link quality; if the packet loss rate exceeds 5%, a link switching operation is triggered. If a drone's CPU utilization exceeds 70% for 10 seconds, it indicates that the drone is overloaded. In this case, the subunit automatically distributes its 10%-20% monitoring area to neighboring drones to balance the load.

[0041] The master unit periodically (every 5-10 minutes) generates a cluster health report, which includes information such as endurance prediction and equipment failure warnings. For slave units that need charging, the master unit plans a return route, raising the slave unit's height by 20-30 meters to avoid being above main traffic flow. At the same time, the master unit dispatches backup slave units to fill the monitoring blind spots created by the return of slave units, ensuring the continuity and integrity of monitoring work.

[0042] Multimodal data acquisition module: A thermal imaging camera acquires infrared images, a visible light camera captures color video, a binocular camera generates depth maps, and a 64-256 line LiDAR acquires 3D point clouds. All devices are synchronized via PPS synchronization pulses. In the collaborative acquisition phase, the thermal imaging camera is set to a resolution of 640×512, outputting infrared images at 15fps, enabling temperature measurement from -20℃ to 150℃. The visible light camera has 4K resolution, capturing color video at 25fps for clear visual images. The binocular camera synchronously generates depth maps with a ranging range of 5-50m and an error controlled within ±3cm, accurately sensing object distance information. The LiDAR outputs 20 frames of point cloud per second, with a point cloud density of 200 points / m², providing rich 3D information for environmental modeling. All devices are synchronized via PPS pulses, triggered on the rising edge, with an error not exceeding 50μs. The acquired data carries a GPS timestamp accurate to milliseconds, ensuring consistency in time across all devices.

[0043] Data preprocessing trigger conditions are dynamically adjusted based on different environmental factors. When the drone's flight speed exceeds 60 km / h, high-speed motion may cause image blurring; the system automatically activates motion blur correction to ensure image clarity. When the light intensity is below 500 lux, insufficient light conditions affect image quality; in this case, HDR synthesis mode is activated to enhance image brightness and contrast. In rainy or snowy weather conditions (determined by an image brightness variance greater than 800), LiDAR point cloud data will be affected; the system triggers LiDAR point cloud enhancement to improve the quality and accuracy of the point cloud data.

[0044] For synchronization, the IEEE 1588PTP protocol is used. The main unit's GPS module serves as the master clock, with a timing accuracy of ±30-80ns, while the slave units and various sensors act as slave clocks. Synchronization formula t sync =t GPS +Δt prop +Δt proc Used to calculate the final timestamp (in seconds) of sensor data, where t sync t represents the final timestamp of the sensor data (in seconds). GPS The original GPS time, Δt propThe signal propagation delay (in seconds, ≤2μs, calculated using the Time-of-Flight (TOF) algorithm to determine the distance between the master and slave units) is Δt. proc The sensor's internal processing delay is specified (unit: seconds, ≤80μs, stored in the device register after pre-calibration). During implementation, synchronization calibration is performed every 5-15ms, using a phase-locked loop (PLL) to compensate for clock drift. For LiDAR point clouds and visible light images, timestamp interpolation alignment (interpolation step size: 1ms) is used to ensure that the time difference between multi-sensor data does not exceed 2ms. If a synchronization anomaly occurs, such as GPS signal loss exceeding 30s, the system automatically switches to the internal crystal oscillator (accuracy ±0.1ppm) to maintain synchronization until the GPS signal returns to normal. Through this series of rigorous operating procedures, the multimodal data acquisition module can efficiently and accurately acquire and synchronize various types of data, laying a solid foundation for subsequent data analysis and processing.

[0045] Image preprocessing module: For high-resolution images, bilinear interpolation is used to downsample them to the 1080p-2K range. Bilinear interpolation calculates the target pixel value by weighted averaging of the gray values ​​of four adjacent pixels. This method can reduce resolution and data volume while better preserving the visual effect of the image and reducing the computational burden on subsequent processing.

[0046] The downsampled image was filtered using Gaussian filters with dimensions ranging from 3×3 to 5×5 and a standard deviation σ between 1.0 and 1.5. Gaussian filtering, based on the Gaussian function, assigns greater weight to pixels closer to the target pixel, effectively suppressing random noise caused by sensor errors and environmental interference, thus smoothing the image.

[0047] The CLAHE algorithm is used to enhance image contrast. The image is divided into grid regions ranging from 4×4 to 16×16, and each region is equalized. The cliplimit is set to 1.5-3.0 to limit the peak value of the histogram, avoid noise amplification and distortion, and adaptively enhance the contrast of each region to highlight image details.

[0048] When motion blur is present in the image, dynamic correction is enabled. The formula is used. Where I(x,y) is the pixel value at (x,y) in the blurred observed image, O(x,y) is the corresponding pixel value in the original clear image, v is the UAV's relative ground speed (30-80m / s), t is the time variable, T is the camera exposure time (1 / 100-1 / 30s), and α is the light intensity attenuation coefficient dynamically adjusted between 0.1 and 0.3 according to the ambient brightness. In specific correction, the motion vectors of adjacent frames are first calculated using the Lucas-Kanade optical flow method (window 11×11-19×19), and pixel displacement is determined based on the assumption that pixel brightness remains constant over short time. For linear blur caused by high-speed flight greater than 60km / h, Wiener filtering with a cutoff frequency of 40-60Hz is used for deconvolution. Wiener filtering restores details based on image statistical characteristics, and deconvolution compensates for motion blur. Simultaneously, the Sobel operator is used to calculate horizontal and vertical gradients to enhance edges. Finally, the Tenengrad gradient value is used to evaluate the correction effect; an improvement of 30%-50% is considered effective; otherwise, low-blurry backup images synchronously acquired by the slave device are used for fusion and supplementation. The entire preprocessing process will keep the delay within 10-20ms, balancing processing effectiveness and real-time requirements;

[0049] Projectile Recognition Module: This module is based on an improved YOLOv8-nano to YOLOv8-s network. By adding an attention mechanism, it can focus more on key regions in the image, improving the ability to recognize projectiles. The network input size is set between 416×416 and 800×800. During the training phase, a training set containing 80,000 to 120,000 labeled samples is used, covering 12 to 18 types of projectiles, enabling the model to learn a rich variety of projectile features.

[0050] During the recognition process, the module integrates RGB image and depth map features, fully utilizing visual information such as color and texture, as well as depth information. When the model predicts a target, it outputs a confidence score. The confidence score threshold is set between 0.6 and 0.7. For suspected targets with confidence scores between 0.3 and 0.65, a secondary verification is performed using the DBSCAN algorithm, which uses point cloud clustering. The key parameter ε in the DBSCAN algorithm ranges from 0.3 to 0.5m, and min_samples ranges from 3 to 5. This algorithm can determine whether a suspected target is a real projectile based on the density distribution of the point cloud data, eliminating some false positives.

[0051] In addition, the module incorporates deep confidence calculation to aid in recognition. This is achieved using the formula... Where C d σ represents depth confidence, ranging from 0 to 1, and is used to measure the probability that the current pixel belongs to a foreground object (such as spilled material); dIt is the standard deviation of the current measurement point depth (unit: m, range: 0.02-0.05 m, obtained from the binocular camera calibration parameters), reflecting the fluctuation of the current measurement point depth; d is the standard deviation of the road background depth (in meters, ranging from 0.03 to 0.07 meters, obtained through historical data statistics), reflecting the dispersion of the road background depth; d is the measured depth value of the current point (in meters), that is, the actual depth of the point from the camera; d0 is the mean of the road background depth (in meters, calculated in real time through a sliding window of 50×50 to 100×100 pixels), representing the average depth of the road background.

[0052] In practice, C is calculated for each pixel in the image. d When C d When the value is less than 0.2 to 0.4, the pixel is initially determined to be a foreground target. To further filter out interference such as road shadows and markings, RGB features are used, such as a color histogram difference greater than 0.3 and a texture entropy greater than 5, for further filtering. For areas ultimately identified as spilled material, their smallest bounding rectangle is extracted, requiring a width × height greater than 0.3m × 0.2m, to further eliminate small-sized noise. In this way, the accuracy of depth-assisted recognition can be improved by 10% to 20%. Figure 4 A radar chart for prioritizing spill disposal is used, displaying four scoring dimensions along the polar axis: size, distance from lane, hazard level, and traffic flow. The radius indicates the score (increasing from low to high). The chart clearly shows that hard metal objects score highly across all dimensions (e.g., size and hazard level scores are close to full marks), cardboard boxes score the lowest, and plastic buckets score in the middle. This multi-dimensional radar graphic visually distinguishes the different disposal priorities of various types of spilled materials, providing a visual basis for intelligent decision-making. Through the above series of operations, the spilled material identification module can accurately identify spilled materials on highways, providing a reliable basis for subsequent disposal work.

[0053] 3D Modeling and Localization Module: This module calculates initial 3D coordinates using the principle of binocular parallax. Binocular cameras capture images of the projectile from different positions, resulting in parallax due to differences in viewing angles. This parallax is measured and combined with camera intrinsic parameters (such as focal length and principal point position) and extrinsic parameters (relative position and attitude between cameras) to calculate the coordinates using triangulation. Next, the Iterative Closest Point (ICP) algorithm is used to fuse the LiDAR point cloud data. First, the closest point pair is found between the binocular camera point set and the LiDAR point cloud set. A transformation matrix containing rotation and translation information is calculated to make the point clouds overlap. This process is repeated 10-30 times to ensure the registration error is less than 0.08m.

[0054] Then, a 3D model is constructed based on the fused data. A triangulation algorithm is used to convert the point cloud into triangular patches, and optimization is performed to maintain the number of triangular patches between 300 and 800, meticulously depicting the shape of the projectile. Afterwards, coordinate transformation is performed, using matrix operations such as rotation and translation to convert the model coordinates to the geographic coordinate system, outputting the positioning result with an error of less than 0.3m, accompanied by a timestamp accurate to milliseconds.

[0055] When rain or snow is identified by a camera image brightness variance greater than 500, the environmental adaptive compensation subunit comes into play. For LiDAR point clouds, the reflection intensity of each point is acquired, and only points with a variance greater than 200 are retained to filter noise. For binocular depth maps, the threshold is increased by 10%-20% compared to clear weather, and the depth map data is traversed to remove invalid depth values ​​below the dynamic threshold. When GPS signal loss is less than 60 seconds, INS inertial navigation data is fused. The inertial navigation system continuously outputs position, velocity, and attitude information, which is fused using Kalman filtering. Based on the system state equation and observation equation, Kalman filtering recursively calculates the optimal state estimate for the current moment using the state estimate from the previous moment and the observation value at the current moment, correcting position drift and controlling the error within 0.5m. After the 3D model is generated, the "dangerous volume" of the spilled material protruding more than 0.1m from the road surface is calculated. The height is determined and the volume is accumulated by traversing the triangular facets of the model. When calculating the "lane encroachment ratio", the projection range of the spilled material on the lane plane is determined and divided by the width of the lane to obtain the ratio, providing a quantitative basis for disposal.

[0056] The intelligent decision-making and scheduling module obtains real-time traffic flow data from the highway management platform every 3-10 minutes, covering key information such as traffic flow and average vehicle speed for each road segment. It also determines the initial handling priority based on parameters such as the lane encroachment ratio and dangerous volume of the spilled material. Generally, spilled materials with higher lane encroachment ratios and larger dangerous volumes receive higher priority. The path planning stage employs an improved ant colony algorithm. Based on the traditional algorithm simulating ant foraging behavior, it optimizes the pheromone update strategy and heuristic function. The planning process comprehensively considers traffic flow, spilled material location distribution, and the drone's endurance. Through multiple rounds of iterative calculations, the error in the generated inspection path length is strictly controlled within 8%, ensuring path efficiency. In emergency situations where spilled material severely impacts traffic or poses significant safety hazards, the system immediately triggers a rapid response mechanism within 2-5 minutes, dispatching resources to intervene and handle the situation as quickly as possible.

[0057] The dynamic priority adjustment subunit is responsible for optimizing the handling sequence in real time. Every 30-60 seconds, it reassesses the priority based on changes in the state of the spilled material. When it detects displacement or disintegration of the spilled material due to vehicle collisions, it updates the priority value promptly. To more accurately measure risk, a "risk diffusion index" is introduced. This index is calculated by integrating real-time wind data (from meteorological monitoring equipment along the route), average vehicle speed (from traffic flow data), and the material properties of the spilled material (extracted from the identification results). For example, the risk diffusion index of lightweight spilled material increases significantly in windy weather, and the risk index of spilled material on road sections with higher vehicle speeds also increases accordingly. For emergency targets, the system automatically instructs the drone to switch to tracking and shooting mode, increasing the frame rate to 30-40fps to capture clear dynamic images. Simultaneously, dynamic selection markers are added to the real-time video stream pushed to the management center, using red borders to track the location of the spilled material in real time, allowing managers to intuitively grasp the situation on-site.

[0058] The route planning process employs a "zonal disposal strategy," dividing the highway into independent disposal units at 5-10km intervals. Each unit has its real-time location information for road administration vehicles pre-entered. During dispatch, the system prioritizes dispatching the 2-3 closest road administration vehicles from the unit containing the spilled material. By calculating the straight-line distance between the vehicle's real-time location and the spilled material's coordinates, as well as the estimated travel time, the optimal dispatch vehicles are selected, avoiding time losses caused by cross-regional dispatch and ensuring efficient resource allocation. Through this multi-stage collaborative operation, the intelligent decision-making and dispatch module achieves intelligent management of the entire process from risk assessment to resource allocation, effectively improving the response speed and accuracy of highway spill disposal.

[0059] The coordinated response module relies on a 4G / 5G private network for data transmission, strictly controlling transmission latency to within 150ms to ensure real-time delivery of alarm information and instructions. Once the system identifies spilled material and determines the need for response, it immediately pushes alarm data containing key information such as the location, type, and risk level of the spilled material. Simultaneously, it remotely controls variable message signs within a 1-3km range and sends precise navigation coordinates to the nearest road maintenance vehicle, guiding it to the scene quickly. The entire response process is recorded by the system, including the alarm trigger time, dispatch instructions, arrival time of response personnel, and completion status. The archiving period is set at 0.5-3 years depending on the risk level, providing complete data support for subsequent traceability and analysis.

[0060] The multi-level early warning linkage subunit activates different levels of response mechanisms based on the risk assessment results of the spilled material. Level 1 warnings target low-risk spills, displaying and automatically recording the warning information only on the monitoring platform of the management center. No additional external linkage operations are required, and management personnel can arrange routine inspections and handling according to the actual situation. Level 2 warnings are suitable for medium-risk scenarios. In addition to displaying and recording at the management center, variable message signs within 1-3km are simultaneously triggered, adjusting their flashing frequency to 2-3 times / second to prominently remind passing vehicles to avoid the spill. The distance information of the spilled material displayed on the signs is accurate to ±50m, helping drivers prepare in advance. Level 3 warnings, the highest level of response, target high-risk spills. The system directly links with the highway traffic police system, pushing a handling plan including suggested road closure length (usually 1-3km) and 2-3 alternative routes to assist the traffic police department in quickly deciding on traffic control measures. Simultaneously, drones are dispatched to hover at a height of 100-150m above the spilled material, activating strobe lights at a frequency of 5-10Hz to further enhance on-site safety protection through strong light warnings.

[0061] All warning messages are assigned a unique code, which is used throughout the entire handling process and is linked to data such as spill identification information, dispatch records, and handling results, forming a complete closed-loop management chain. When road administration personnel complete the spill cleanup and report the completion of the handling in the system, the module automatically identifies the handling result, cancels the corresponding warning status, the variable message signs return to normal display, the drone stops its flashing lights and returns to base, and the traffic police system also receives a synchronous notification of completion, allowing for timely adjustments to traffic control measures. Through the association effect of the unique code, each warning message corresponds to a clear handling result, ensuring that the entire handling process is traceable and verifiable, achieving a closed-loop record from warning triggering to handling completion.

[0062] In this invention, the projectile recognition module further includes a few-shot learning enhancement subunit: dedicated to overcoming the challenge of recognizing rare and unknown types of projectiles. For rare projectiles with fewer than 500 training samples, transfer learning is performed using an ImageNet pre-trained model. The ImageNet pre-trained model has learned general features from a large amount of image data, which can quickly provide basic features for projectile recognition. Simultaneously, data augmentation techniques are used to randomly rotate the original samples (±15°) to simulate imaging from different perspectives; scale them (0.8-1.2 times) to adapt to changes in projectile size at different shooting distances; and add Gaussian noise to simulate actual shooting noise interference, expanding the sample size to 2000-3000 images to enrich the model's learning materials.

[0063] A contrastive learning mechanism is introduced, constructing positive and negative sample pairs of "spillage-background" at a 1:3 ratio, and using a Siamese network to learn discriminative features. The Siamese network consists of two subnetworks with identical structures and weights, which are respectively input with positive and negative samples. Through a specific loss function, the features of positive samples are made closer together and the features of negative samples are made further apart, thereby strengthening the model's ability to discriminate spillage.

[0064] When encountering new, unknown types of spilled material with a confidence level <0.3 and not belonging to a known category, it is automatically marked as "to be classified" and sent to manual review. After review and labeling by the reviewers, the results are incorporated into the training set, and the model is incrementally updated once a week to allow the model to continuously learn new features and improve the accuracy of identifying various types of spilled material in order to cope with complex real-world scenarios.

[0065] In this invention, the image preprocessing module further includes an adaptive illumination adjustment subunit: In backlit scenes (contrast ratio > 100:1), HDR compositing technology is used to acquire 3-5 images of the same scene with varying exposures, which are then fused using a specific algorithm to balance details in both bright and dark areas. Next, the Retinex algorithm is applied to separate the illumination and reflection components, and the Gamma value (0.5-2.0) is dynamically adjusted based on the results. If the image is too dark, Gamma is increased to brighten it; if it is too bright, Gamma is decreased to darken it, further enhancing image contrast and clarity.

[0066] In nighttime scenes (average brightness < 50), thermal imaging and visible light image fusion are initiated, with weights adjusted according to the actual situation (7:3-5:5). Thermal imaging can highlight the infrared characteristics of high-temperature debris (such as tire fragments), facilitating identification. Simultaneously, an adaptive threshold segmentation algorithm is employed to determine the segmentation threshold based on local image features, separating the strong headlight glare from other areas. Morphological filtering is then used, employing operations such as erosion and dilation to remove strong light interference noise and suppress the influence of strong headlight glare. This processing achieves all-weather recognition stability, controlling the difference in recognition accuracy between day and night to < 10%. Figure 3 The graph shows a comparison of the accuracy of this system and the traditional system in identifying spilled objects under different environments. The horizontal axis represents four environmental types: sunny, cloudy, rainy, and backlit. The vertical axis represents the accuracy rate (%). This visually verifies the advantages of this system in identifying spilled objects in complex environments through technologies such as adaptive light adjustment.

[0067] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automatic identification system for highway spills based on unmanned aerial vehicles (UAVs), characterized in that, Includes the following modules: The drone swarm deployment module consists of one mother drone and 3 to 5 drones. It adopts a Mesh self-organizing network. The drones are distributed in a diamond array, covering a lane with a width of 150 to 250 meters. The mother drone cruises at an altitude of 120 to 180 meters, and the drones fly at an altitude of 80 to 120 meters. All drones are equipped with RTK-GPS. Multimodal data acquisition module: Deploys a thermal imaging camera to acquire infrared images, a visible light camera to capture color video, a binocular camera to generate depth maps, and a 64-256 line lidar to acquire 3D point clouds. All devices achieve time synchronization through PPS synchronization pulses. Image preprocessing module: High-resolution images are downsampled to 1080p~2K using bilinear interpolation, noise is removed by 3×3~5×5 Gaussian filtering, contrast is enhanced by CLAHE algorithm, and dynamic correction is enabled for motion-blurred images. The projectile identification module is based on the improved YOLOv8-nano to YOLOv8-s network, which integrates RGB image and depth map features. The training set contains 80,000 to 120,000 labeled samples, and the confidence threshold is set to 0.6 to 0.

7. Suspected targets are verified by point cloud clustering. 3D modeling and positioning module: The initial 3D coordinates are calculated using the binocular parallax principle. The ICP algorithm is used to fuse the point cloud with the LiDAR to construct a 3D model of the projectile. The positioning result is output after coordinate transformation, with a timestamp attached. Intelligent decision-making and scheduling module: It obtains real-time traffic flow from the highway management platform, generates disposal priorities by combining spilled material parameters, plans inspection paths using an improved ant colony algorithm, and triggers a response mechanism within 2 to 5 minutes for emergencies; Joint response module: Pushes alarm information through 4G / 5G private network, simultaneously controls variable message signs within 1-3km, sends navigation coordinates to the nearest road administration vehicle, and records the entire response process.

2. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, It also includes the motion blur correction algorithm in the image preprocessing module: Where I(x,y) represents the pixel value of the blurred observed image, O(x,y) represents the pixel value of the original clear image, v represents the UAV's relative flight speed to the ground, T represents the camera exposure time, t represents the time variable, and α represents the light intensity attenuation coefficient. In practice, the motion vectors of adjacent frames are first calculated using the Lucas-Kanade optical flow method. For linear blurring caused by high-speed flight, Wiener filtering is used for deconvolution, and gradient enhancement algorithm is used simultaneously to strengthen edge features. After correction, the image quality is evaluated. When the evaluation value is improved by ≥30% to 50%, the correction is considered effective; otherwise, a backup image is used for fusion and supplementation.

3. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, It also includes the depth confidence calculation in the spill identification module: Among them, C d For depth confidence, σ d The standard deviation of the depth at the current measurement point. Let d be the standard deviation of the road surface background depth, d be the measured depth value at the current point, and d0 be the mean of the road surface background depth. During implementation, C is calculated for each pixel in the image. d When C d When the value is <0.2 to 0.4, it is initially identified as a foreground target. Then, the road surface interference is filtered out by combining RGB features. For areas identified as spilled objects, the minimum bounding rectangle is extracted to eliminate small-sized noise.

4. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The synchronization mechanism of the multimodal data acquisition module is as follows: time synchronization is achieved using the IEEE 1588PTP protocol. The master clock is the mother unit's GPS module, and the slave clocks are the slave units and each sensor. The synchronization formula is: t sync =t GPS +Δt prop +Δt proc Among them, t sync t is the final timestamp of the sensor data. GPS The original GPS time, Δt prop For signal propagation delay, Δt proc To compensate for the internal processing delay of the sensor, during implementation, a synchronization calibration is performed every 5 to 15 ms. Clock drift is compensated by a phase-locked loop, and timestamp interpolation is used to align the lidar point cloud and the visible light image. In case of synchronization failure, the system automatically switches to the internal crystal oscillator of the host machine to maintain synchronization until the signal is restored.

5. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The intelligent decision-making and scheduling module also includes a dynamic priority adjustment subunit: based on the real-time status changes of the spilled material, the priority is reassessed every 30 to 60 seconds, and a "risk diffusion index" is introduced; For emergency targets, drones are automatically triggered to track and capture images, and the video streams are simultaneously pushed to the management center with dynamic bounding boxes and annotations. When planning routes, a "zonal handling strategy" is adopted, dividing highways into handling units of 5-10km. Each unit prioritizes dispatching the nearest 2-3 road administration vehicles to avoid efficiency losses from cross-regional dispatching.

6. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The drone swarm deployment module also includes an adaptive load balancing subunit: it monitors the battery level and data transmission link quality of each drone in real time, and automatically allocates the monitoring area it is responsible for to neighboring drones when the load of a drone is too high. The master unit periodically generates cluster health reports, plans return routes for slave units that need charging, and simultaneously dispatches backup slave units to fill monitoring blind spots.

7. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The linkage response module also includes a multi-level early warning linkage sub-unit: Level 1 early warning is only displayed and recorded in the management center; Level 2 early warning simultaneously triggers variable message signs within 1-3km; Level 3 early warning links with the highway traffic police system, pushes road closure suggestions, and simultaneously controls the drone to hover 100-150m above the dumped material and activates flashing lights to warn passing vehicles. All early warning information is accompanied by a unique code, and the associated handling results form a closed-loop record.

8. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The projectile identification module also includes a few-shot learning enhancement subunit: for rare projectile types, transfer learning combined with data augmentation techniques is used to expand the sample to 2000-3000 images; a contrastive learning mechanism is introduced to construct "projectile-background" positive and negative sample pairs, and discriminative features are learned through a Siamese network; For newly emerging unknown types, they are automatically marked as "to be classified" and sent to manual review. The review results are then incorporated into the training set to achieve incremental updates of the model.

9. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The 3D modeling and positioning module also includes an environmental adaptability compensation subunit: for rainy and snowy weather, noise filtering is performed on the lidar point cloud, and invalid values ​​are eliminated by dynamic thresholding of the binocular depth map; during positioning, INS inertial navigation data is fused, and position drift is corrected by Kalman filtering; after the 3D model is generated, the "dangerous volume" and "lane encroachment ratio" of the spilled material are automatically calculated to provide a quantitative basis for disposal.

10. The automatic identification system for highway spills based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The image preprocessing module also includes an illumination adaptive adjustment subunit: it uses HDR synthesis technology to process backlight scenes, separates the illumination component and reflection component of the image through the Retinex algorithm, and dynamically adjusts the Gamma value. For nighttime scenes, thermal imaging and visible light image fusion are used to highlight the infrared characteristics of high-temperature debris while suppressing the glare from vehicle headlights, ensuring all-weather recognition stability.

Images