Roadside camera-based road litter detection method, device, medium and product

Abstract

The embodiment of the application relates to the field of intelligent transportation, and discloses a road litter detection method and device based on a roadside camera, a medium and a product; the method comprises the following steps: acquiring a real-time road video stream collected by a roadside camera, and extracting a single frame image; dividing the single frame image into multiple detection areas, and setting a corresponding slice window for each detection area for sampling; wherein the size of the slice window of the detection area close to the top of the single frame image is smaller than the size of the slice window of the detection area close to the bottom of the single frame image; inputting the image sampled by the slice window into a pre-tuned visual language large model to output a target detection result; mapping the target detection result back to the coordinate system of the single frame image based on the position coordinates of the slice window in the single frame image to obtain a suspected litter detection result of the single frame image, so as to detect road litter.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation, and in particular to a method, device, medium and product for detecting road spills based on a roadside camera. Background Technology

[0002] With the rapid development of intelligent transportation systems, road safety has become increasingly important. Timely detection and cleanup of road debris (such as cargo dropped from trucks, vehicle fragments, and irregularly piled debris) is a crucial aspect of preventing traffic accidents. Current road debris detection primarily relies on traditional pure visual deep learning models (such as the YOLO series algorithms) or traditional image processing algorithms based on background modeling. However, in practical roadside monitoring applications, these traditional solutions face several insurmountable bottlenecks.

[0003] First, traditional object detection algorithms operate on closed-set detection, heavily relying on large-scale labeled training data and requiring consistent category distributions between the training and test sets. However, in road litter scenarios, positive samples (images containing litter) have an extremely low probability of appearing in massive amounts of surveillance video, representing typical long-tailed data. This makes it difficult to collect sufficient and diverse positive samples to train the model, easily leading to overfitting or insufficient generalization. Second, litter has a wide variety of physical forms, encompassing countless possible objects such as cardboard boxes, stones, metal parts, and tires. Traditional algorithms attempt to classify and train by exhaustively listing all litter categories, resulting in extremely high labeling costs. Furthermore, when faced with unknown litter categories not present in the training set, the model often fails to identify them, leading to missed detections.

[0004] Furthermore, roadside cameras are typically installed at a high position with a fixed overhead view, resulting in a significant perspective effect in the imaging, where objects appear larger in the foreground and smaller in the background. Debris located at the far end of the image often occupies only a very small number of pixels (e.g., less than 10×10 pixels), making its features extremely blurry. Traditional full-image input detection or uniform slice detection methods struggle to balance the feature scale of large nearby targets with extremely small distant targets, leading to very low recall rates for these small distant targets. Simultaneously, the complex background of roadside scenes, including road markings, tree shadows, and distant vehicles, makes them highly susceptible to misidentification as debris. The lack of an effective spatiotemporal logic verification mechanism to eliminate these interferences results in a high false alarm rate for existing systems, making them unsuitable for practical engineering deployments. Summary of the Invention

[0005] One objective of this application is to provide a method, device, medium, and product for detecting road debris based on a roadside camera, at least to solve the problems in existing road debris detection technologies, such as the scarcity of positive sample data leading to difficulties in model training, the complexity of debris categories making exhaustive coverage difficult, and the low recall rate and high false alarm rate of detecting extremely small targets at a distance from the roadside perspective.

[0006] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0007] In a first aspect, some embodiments of this application also provide a method for detecting road debris based on a roadside camera, the method comprising:

[0008] Acquire real-time road video streams captured by roadside cameras and extract single-frame images;

[0009] The single-frame image is divided into multiple detection regions, and a corresponding slice window is set for each detection region for sampling; wherein, the slice window size of the detection region located near the top of the single-frame image is smaller than the slice window size of the detection region located near the bottom of the single-frame image;

[0010] The image sampled from the slice window is input into a pre-tuned large visual language model, and the target detection result is output.

[0011] Based on the position coordinates of the slice window in the single-frame image, the target detection result is mapped back to the coordinate system of the single-frame image to obtain the suspected spill detection result of the single-frame image.

[0012] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0013] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the method described above.

[0014] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0015] Compared with related technologies, the solution provided in this application firstly, by introducing a Visual Language Model (VLM) and combining it with an open set detection mechanism, fundamentally breaks through the dependence of traditional object detection algorithms on training data of specific categories. By utilizing the massive pre-trained knowledge and natural language instruction-driven capabilities of VLM, it can identify various unknown debris not appearing in the training set, effectively solving the industry problems of the inexhaustible types of debris and the extreme scarcity of positive sample data, and significantly reducing the cost of data collection and annotation.

[0016] Secondly, by dividing the image into detection regions with different viewing distance attributes and adopting a differential sampling and overlapping strategy of small windows for distant views and large windows for near views, small objects in the distance are relatively magnified in the sliced ​​image, thereby obtaining feature resolution similar to that of near objects when inputting into the model. This effectively solves the problem of missed detection of small distant targets caused by perspective effect and significantly improves the detection recall rate of the entire road surface.

[0017] Finally, on the one hand, by introducing a difficult example dataset containing distractors such as traffic markings and shadows during the training phase, and combining it with a freeze-layer fine-tuning strategy, false detections of static backgrounds by the model were suppressed. On the other hand, a spatiotemporal verification mechanism based on multi-frame position offsets was introduced during the inference phase, utilizing the static physical properties of debris to accurately eliminate moving distracting targets such as distant vehicles moving along the lane. This dual filtering mechanism ensures high confidence and robustness of the detection results. Attached Figure Description

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 An exemplary flowchart of a road spill detection method based on a roadside camera, provided for some embodiments of this application;

[0020] Figure 2 A schematic diagram illustrating the division of a road debris detection method based on a roadside camera, provided for some embodiments of this application;

[0021] Figure 3 An exemplary structural diagram of the electronic device provided for some embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The following terms are used in this document.

[0024] VLM (Vision-Language Model): A large-scale visual language model, which is an artificial intelligence model that integrates visual and linguistic information for processing.

[0025] Grounding DINO: An open-source visual language object detection model based on the Transformer architecture;

[0026] GLIP (Grounding Language-Image Pre-training): A visual language model that achieves grounding capability through language-image pre-training.

[0027] Figure 1 An exemplary flowchart of a road spill detection method based on a roadside camera, provided for some embodiments of this application, is shown below. The method includes:

[0028] S101. Acquire the real-time road video stream collected by the roadside camera and extract single-frame images.

[0029] Specifically, step S101 is executed first to acquire the real-time road video stream collected by the roadside camera and extract single-frame images. In practical applications, the roadside camera connects to the cloud control platform via a 4G or 5G network. Considering the balance between bandwidth and computing resources, the acquisition frequency can be set to 1 to 5 frames per second; when a large traffic flow is detected, it can be automatically adjusted to 5 frames per second to ensure sampling density. The acquired video stream is decoded into single-frame images, and the resolution is uniformly adjusted to a standard size (e.g., 1920×1080 pixels) to ensure consistency in subsequent processing.

[0030] S102. Divide the single-frame image into multiple detection regions; and set a corresponding slice window for sampling for each detection region; wherein, the slice window size of the detection region located near the top of the single-frame image is smaller than the slice window size of the detection region located near the bottom of the single-frame image.

[0031] Specifically, in step S102, the single-frame image is divided into multiple detection regions, and a corresponding slice window is set for sampling in each region. Considering the high-position, top-down installation characteristics of roadside cameras, the image exhibits a significant perspective effect. Therefore, the image can be divided vertically, and the number of divisions is configurable. For example, the lower half of the image can be designated as the near-field region, and the upper half as the far-field region. A larger slice window (e.g., 512×512 pixels) is used for sliding sampling in the near-field region to match the size of larger nearby targets; while a smaller slice window (e.g., 256×256 pixels) is used for sliding sampling in the far-field region to relatively magnify smaller distant targets, ensuring their feature resolution meets the model's detection requirements. Alternatively, as shown... Figure 2 As shown, the image is divided vertically into three levels of detection regions: near-field, mid-field, and far-field. For example, the bottom third of the image is designated as the near-field region, using a slice window of the first size (e.g., 640×640 pixels); the middle third is designated as the mid-field region, using a slice window of the second size (e.g., 320×320 pixels); and the top third, near the vanishing point, is designated as the far-field region, using a slice window of the third size (e.g., 160×160 pixels). This sampling method, with progressively smaller window sizes from near to far, ensures that road debris at different depths occupies a relatively consistent pixel proportion in each slice image, thus significantly improving the model's detection stability for targets in mid- to far-distance transition areas.

[0032] S103. Input the image sampled by the slice window into the pre-fine-tuned visual language large model and output the target detection result.

[0033] Specifically, step S103 is then executed, inputting the image sampled from the slice window into a pre-tuned Visual Language Model (VLM) and outputting the target detection result. The Visual Language Model can be a model with open-set detection capabilities, such as Grounding DINO or GLIP. It contains sufficient positive sample features of "similar litter," supporting open-set detection without exhaustively listing litter types. The fine-tuning dataset can collect ≥500 actual litter data images, covering different weather conditions, time periods, and litter types, and is used to pre-train the model. During inference, a natural language instruction described as "litter on the road" is input, driving the model to output the target category and local coordinates within each slice.

[0034] S104. Based on the position coordinates of the slice window in the single-frame image, the target detection result is mapped back to the coordinate system of the single-frame image to obtain the suspected debris detection result of the single-frame image.

[0035] Specifically, in step S104, based on the position coordinates of the slice window in the single-frame image, the target detection result is mapped back to the coordinate system of the single-frame image to obtain the suspected debris detection result of the single-frame image. The system records the coordinate offset of the upper left corner of each slice window in the original image, adds the local coordinates output by the model to the offset, calculates the global coordinates of the target, and thus generates the final suspected debris detection result of the single-frame image.

[0036] In this embodiment, the introduction of a large visual language model effectively solves the problems of scarce positive samples and inexhaustible categories in traditional detection methods for road debris, achieving high-precision open-set detection of debris of unknown categories. Furthermore, based on a differentiated slicing sampling mechanism, the feature loss caused by perspective projection is compensated for at the algorithmic level, ensuring that extremely small targets at a distance have sufficient resolution features when input into the model, significantly improving the detection recall rate of small debris across the entire road surface, especially at long distances.

[0037] In one embodiment, after obtaining the suspected debris detection result of the single-frame image, the method further includes performing a spatiotemporal consistency check on the suspected debris detection result, specifically including:

[0038] Within a set time window, extract the detection results of the suspected spilled material from multiple consecutive frames of images;

[0039] Calculate the positional offset of the suspected spill detection result between the consecutive multiple frames of images;

[0040] If the positional offset is less than a preset static threshold, it is identified as road debris.

[0041] If the position offset is greater than the stationary threshold, it is identified as a moving interference target and removed from the detection results.

[0042] Specifically, after the system outputs the suspected debris detection results of a single frame image using the visual language large model, it performs a spatiotemporal consistency verification process to further eliminate dynamic interference in the environment. The system first sets a verification time window (e.g., a video segment lasting 1 to 2 seconds) and extracts the detection results of multiple consecutive frames within this time window from the video stream buffer. The system uses target tracking algorithms (such as IoU-based matching or Kalman filtering) to associate the same detected target in consecutive frames, forming a target motion trajectory. Subsequently, the system calculates the positional offset of the suspected debris between the consecutive frames. This positional offset is typically calculated as the Euclidean distance between the center point of the target detection box and the beginning and end frames of the time window, or the cumulative displacement distance between adjacent frames. Next, the system compares the calculated positional offset with a preset static threshold (e.g., 5 to 10 pixels, which can be adaptively adjusted according to the camera's installation height and resolution). If the positional offset is less than the preset static threshold, it indicates that the target has maintained a fixed position on the road surface for a long time, and the system confirms it as genuine road debris and generates an alarm event. Conversely, if the position offset is greater than the stationary threshold, regardless of its direction of movement, the system determines that the target is a moving interference target (which may include distant vehicles, birds crossing the road, pedestrians, or non-fixed objects being blown by the wind) and removes it from the list of suspected detection results, thereby ensuring the accuracy of the final output result.

[0043] In this embodiment, by introducing a spatiotemporal consistency verification mechanism based on displacement thresholds, the anti-interference capability and robustness of the road debris detection system are greatly improved. Utilizing the stationary physical characteristics of the debris, by continuously analyzing the target's motion state across multiple frames, stationary debris can be accurately distinguished from any form of moving interference source. This ensures a high recall rate while significantly reducing the system's false alarm rate and greatly reducing the workload of monitoring personnel in unnecessary verification.

[0044] In one embodiment, the step of setting a corresponding slice window for sampling for each detection region specifically includes:

[0045] Within the same detection area, adjacent slice windows have overlapping areas;

[0046] The overlap rate between the slice windows is negatively correlated with the size of the slice windows.

[0047] Specifically, such as Figure 2As shown, when performing sliding window sampling within the same detection area, adjacent slice windows are not seamlessly stitched together, but rather retain a certain pixel overlap area. Following the configuration principle that the overlap rate between slice windows is negatively correlated with the slice window size, differentiated settings can be made based on the physical viewing distance corresponding to the detection area. For example, in distant areas corresponding to a relatively far physical viewing distance (i.e., the top part of a single frame image), because a smaller slice window size (e.g., 256×256 pixels) is used, and the debris targets in this area occupy very few pixels in the image (often only a few dozen pixels), if the target happens to be located on the segmentation boundary, it is very easy to miss detection due to feature destruction during segmentation. Therefore, the system uses a higher slice overlap rate (e.g., 25% to 30%) in this area, i.e., reducing the step size of the sliding window, and increasing the sampling density to ensure that small distant targets remain relatively intact at least in one slice. In the near-field region (i.e., near the bottom of a single frame image) corresponding to a closer physical viewing distance, a larger slice window (e.g., 512×512 pixels) is used. The target image size in this region is large and rich in features, making it easier to identify even if edge truncation occurs. Therefore, the system uses a lower slice overlap rate (e.g., 10% to 15%) in this region and reduces the total number of generated slices by increasing the sliding step size.

[0048] In this embodiment, an adaptive sampling mechanism is established where the slice overlap rate is negatively correlated with the window size. A high overlap rate is configured for distant regions, effectively mitigating the risk of feature fragmentation or loss caused by truncation of distant, small targets at slice boundaries. This significantly improves the detection recall rate for targets with low pixel counts, ensuring the integrity of the evidence chain. A low overlap rate is configured for near-field regions, effectively suppressing redundant growth in the total number of slices while maintaining the integrity of large-scale target features. This significantly reduces computational resource consumption and network bandwidth usage during the large-scale visual language model inference process, achieving optimal allocation of system computing resources and real-time response.

[0049] In one embodiment, prior to the step of dividing the single-frame image into multiple detection regions, the method further includes:

[0050] The single-frame image is identified using a semantic segmentation model or a large visual language model to generate a road mask that only contains the road surface area.

[0051] Multiple detection areas are divided only within the image area covered by the road mask.

[0052] Specifically, to further improve the system's processing efficiency and reduce false alarms introduced by environmental background, the system pre-introduces a region of interest (ROI) extraction step before performing image block sampling. By loading a pre-trained lightweight semantic segmentation model (e.g., Seg Former, DeepLabV3+) or a general visual language model, full-image semantic analysis is performed on the extracted single-frame image. This model is trained to identify and segment "drivable areas" and "non-drivable areas," thereby generating a binary road mask with the same size as the original image. In this mask, the road surface area is marked as valid pixels (e.g., value 1), while background areas such as trees, buildings, sky, and sidewalks are marked as invalid pixels (e.g., value 0). Subsequently, when executing the sliding sampling logic for the slice window, the system will only generate slices within the valid pixel area covered by the road mask; slice windows completely outside the mask or with a valid pixel ratio below a preset threshold will be skipped and not input into the visual language model for subsequent inference.

[0053] In this embodiment, by introducing semantic segmentation to generate a road mask, a large number of non-road background areas (such as the sky, green belts, buildings, etc.) in the video footage can be accurately removed from the spatial dimension. This allows subsequent slice sampling and large-scale model inference processes to focus only on the actual road surface, thereby avoiding invalid calculations for irrelevant background areas and significantly reducing the overall computing power and processing latency of the system. At the same time, this mechanism blocks the entry of interference from non-road areas at the source, effectively preventing debris on the roadside lawn, birds in the sky, or roadside facilities from being misdetected as road debris by the visual language large-scale model, significantly reducing the false alarm rate of the system and improving the business usability of the detection results.

[0054] In one embodiment, the step of dividing the single-frame image into multiple detection regions further includes:

[0055] Identify the vanishing point or horizon position of the road lanes in the single-frame image;

[0056] Based on the vanishing point or horizon position, a perspective depth gradient model is constructed along the vertical direction of the image;

[0057] Based on the perspective depth gradient model, the ground sampling distance corresponding to pixels at different vertical coordinates in the image is calculated, and the boundary line of the multiple detection areas is determined according to the numerical distribution of the ground sampling distance.

[0058] Specifically, to adapt to complex scenarios where roadside cameras are installed at varying heights or the pan-tilt angle changes dynamically, the system abandons the traditional method of dividing the detection area based on fixed pixel coordinates and instead adopts a dynamic segmentation strategy based on perspective depth gradient. The system first uses lane line detection algorithms (such as edge detection based on Hough transform or lane segmentation models based on deep learning) to identify lane line features in a single frame image and calculates the intersection of lane line extensions on the image plane to determine the vanishing point, or identifies the boundary between the sky and the ground through image segmentation to determine the horizon line. Based on this, the system constructs a perspective depth gradient model distributed along the vertical coordinate axis of the image, according to the principle of perspective projection imaging. This model maps the non-linear relationship between different vertical coordinates (i.e., different rows of pixels) in the image and the actual physical distance to the ground. The system further uses this model to calculate the ground sampling distance (GSD) corresponding to each row of pixels in the image, that is, the actual physical length represented by a single pixel on the road surface (e.g., the GSD of pixels at the bottom of the image is approximately 0.5 cm / pixel, while the GSD of pixels near the vanishing point may reach 10 cm / pixel). Finally, the system locates the corresponding vertical coordinate point on the GSD value distribution curve based on a preset physical resolution threshold (e.g., setting the GSD threshold to 5 cm / pixel), and confirms this coordinate point as the dynamic boundary between the near and far areas. In this way, regardless of how the camera's tilt angle is adjusted, the boundary line will automatically shift as the vanishing point moves, ensuring that the physical scale characteristics of the images on both sides of the boundary line always meet the algorithm design requirements.

[0059] In this embodiment, by introducing perspective depth gradient modeling technology based on vanishing point or horizon, the detection area division and the camera's physical installation parameters (such as height and pitch angle) are successfully decoupled. This means that the algorithm does not require manual on-site measurement of installation data or reading real-time angle feedback from the gimbal; it can adaptively calculate the optimal block boundaries that conform to physical scale entirely based on visual image content. This purely visual self-calibration mechanism greatly enhances the system's generalization ability and engineering applicability, enabling it to perfectly adapt to fixed cameras, motorized zoom cameras, and even mobile inspection equipment. It significantly reduces the system's deployment complexity and subsequent maintenance costs, while ensuring the stability and accuracy of the detection algorithm when the device's posture changes.

[0060] In one embodiment, after the step of mapping the target detection result back to the coordinate system of the single-frame image to obtain the suspected debris detection result of the single-frame image, the method further includes:

[0061] Perform pairwise traversal of all target detection boxes mapped back to the coordinate system of the single-frame image;

[0062] Calculate the intersection-union ratio (CUP) between each pair of object detection boxes;

[0063] If the cross-union ratio is greater than the preset fusion threshold, it is determined to be a duplicate detection, and the target detection box with the higher confidence score is retained, or the target detection boxes of the duplicate detection are weighted and averaged based on the confidence score to be merged into a single target detection box.

[0064] Specifically, after the visual language model completes inference on each slice image and maps all local detection results back to the global coordinate system of a single frame image, the system initiates a fusion and deduplication process targeting overlapping areas. Because an overlapping area (e.g., 15%-20%) is intentionally reserved between adjacent slice windows in the previous steps to prevent target truncation, the same litter target located within this area will inevitably be detected simultaneously by two or more adjacent slice windows, resulting in multiple overlapping redundant detection boxes at the same location in the global image. To eliminate this redundancy, the system first performs pairwise traversal of all mapped candidate target detection boxes and calculates the intersection-over-union (IoU) ratio between each pair of detection boxes, i.e., the ratio of the intersection area to the union area of ​​two detection boxes. Subsequently, the system compares the calculated IoU value with a preset fusion threshold (e.g., 0.5). If the IoU value is greater than this threshold, the system determines that the two detection boxes belong to repeated detections of the same physical target. Based on this, the system performs fusion operations according to preset strategies: one approach is to use non-maximum suppression (NMS) logic to directly compare the confidence scores of two detection boxes, retaining the box with the higher score and removing the box with the lower score; another more preferred approach is to use weighted box fusion (WBF) strategy, which uses the confidence score as the weight to calculate the weighted average of the coordinate positions of all duplicate detection boxes, thereby synthesizing a new coordinate box as the final result.

[0065] In this embodiment, by implementing an IoU-based fusion deduplication strategy, the problem of repeated target detection introduced by the sliding window slicing mechanism is effectively solved, ensuring the uniqueness and accuracy of the final output result and avoiding false alarms caused by the same debris being reported multiple times. Specifically, by employing a confidence-based weighted average fusion method, this application not only removes redundant boxes but also fully utilizes multi-view observation data to perform secondary calibration of the target location. Compared to simply retaining high-resolution boxes, weighted fusion can integrate the prediction information of multiple slices, smoothing out the random errors of single predictions, thereby generating more accurate target bounding boxes than single-model inference, significantly improving the spatial accuracy of road debris localization.

[0066] In one embodiment, the large visual language model is fine-tuned using a strategy that includes hard example mining:

[0067] Construct a difficult example dataset that includes road traffic markings, tree shadows, and roadside facilities;

[0068] The aforementioned difficult example dataset is added to the training data to fine-tune the pre-trained visual language large model.

[0069] Specifically, to reduce the false alarm rate of the system under complex road surface backgrounds, this application introduces a hard example mining strategy during the model training phase to fine-tune the large visual language model. First, a dedicated "hard example dataset" is constructed based on historical false alarm data and manual screening. This dataset mainly contains non-road debris samples that are visually extremely similar to road debris and easily cause model confusion. Specifically, it covers discontinuous white traffic markings on the road surface (easily misjudged as scraps of paper or long strips), dappled shadows cast by trees or buildings (easily misjudged as dark stones), and fixed facilities such as manhole covers and square electrical boxes on the roadside. Subsequently, the system mixes the hard example dataset with positive sample data containing real road debris at a preset ratio (e.g., 1:3) to generate the final fine-tuned training set. During fine-tuning training of the pre-trained large visual language model, negative sample reinforcement training forces the model to focus on and learn to distinguish subtle texture and edge differences between real road debris and hard example interference objects during backpropagation.

[0070] In this embodiment, by constructing a difficult example dataset containing traffic markings, shadows, and facilities and implementing targeted fine-tuning, the visual language model's ability to withstand interference from complex road conditions is significantly enhanced. This enables it to fundamentally identify false targets that closely resemble debris, effectively suppressing false detections caused by changes in lighting (such as shadow movement) or inherent road surface features (such as old markings). This not only significantly reduces the system's false alarm rate and ensures high confidence in the detection results, but also avoids a decline in model generalization ability due to overfitting to simple samples, ensuring the algorithm's robustness in different road environments.

[0071] In one embodiment, the specific strategy for fine-tuning the training is as follows:

[0072] The backbone network parameters of the large visual language model are frozen, and only the parameters of the detection head are updated, while the learning rate is reduced during the fine-tuning process.

[0073] Specifically, to preserve the general cognitive capabilities of the visual language model while efficiently adapting it to the specific scenario of road litter detection, the system employs a training strategy based on Parametric Efficient Fine-Tuning (PEFT). During fine-tuning, the system first freezes the backbone parameters of the visual language model, keeping them unchanged throughout training, and only unlocks and updates the weight parameters of the detection head. Simultaneously, to fully utilize the performance of the pre-trained model and avoid "catastrophic forgetting," this embodiment employs a differentiated hyperparameter configuration strategy: the learning rate during fine-tuning is set significantly lower than that in the pre-training stage (e.g., reduced to 1 / 10 or 1 / 100 of the original), and the number of iterations and regularization parameters are correspondingly reduced to achieve a balance between rapid convergence and preservation of prior knowledge. Furthermore, to find the optimal training configuration for the road litter scenario, the system introduces an automated hyperparameter optimization (HPO) strategy. Utilizing intelligent optimization methods such as genetic algorithms, random search, or grid search, the system performs multiple rounds of trial and error and evaluation on key hyperparameters such as learning rate, regularization parameter, batch size, and number of iterations within a pre-defined search space, thereby calculating the globally optimal combination of hyperparameters. Finally, under the control of the optimal hyperparameters, the system receives natural language commands such as "detect obstacles or debris on the road surface" to guide the model in completing targeted fine-tuning.

[0074] In this embodiment, by freezing the backbone network and reducing the learning rate and number of iterations, the computational cost and time expenditure of model training are significantly reduced. Simultaneously, it effectively prevents the model from losing its original general visual understanding ability due to overtraining on specific small sample data (scattered debris), thus avoiding catastrophic forgetting. Furthermore, by introducing intelligent hyperparameter optimization strategies such as genetic algorithms, the inefficient traditional method of manually tuning parameters based on experience is abandoned. This approach automatically searches for the hyperparameter combination that best matches the distribution characteristics of road litter data using mathematical methods, thereby maximizing the model's performance potential and significantly improving its generalization ability and detection accuracy in complex road scenarios.

[0075] In one embodiment, before inputting the image sampled from the sliced ​​window into the pre-fine-tuned large visual language model, the method further includes:

[0076] Determine whether the size of the sliced ​​image is smaller than the standard input size of the large visual language model;

[0077] If the size is smaller than the standard input size, then fill the edges of the sliced ​​image with non-information pixels until the size matches the standard input size, so as to keep the original pixel resolution of the target in the sliced ​​image unchanged.

[0078] Specifically, to adapt to the fixed input tensor dimension of the large visual language model and avoid blurring of distant small target features due to image scaling, the system incorporates an image size adaptive padding preprocessing step before model inference. After generating slice images with different viewing distance attributes according to the aforementioned steps, the system first reads the resolution of the slice image (e.g., 256×256 pixels for distant slices) and compares it with the standard input size of the large visual language model (e.g., 640×640 pixels). If the slice image size is determined to be smaller than the model's standard input size, the system will not use traditional bilinear or bicubic interpolation algorithms to enlarge and stretch the image, but instead will perform edge padding. The system will fill the edges of the slice image (e.g., the right and bottom edges, or uniformly around the edges) with non-information pixels (e.g., black pixels with an RGB value of 0 or specific mean gray pixels) until the total size of the filled image is completely consistent with the model's standard input size. During this process, the pixel data within the original slice remains absolutely static and preserved as is, without any resampling or geometric deformation. The filled image is then input into the model for feature extraction.

[0079] In this embodiment, by employing a fill-without-stretching image preprocessing strategy, the pixel distortion problem that may be faced when adapting small distant slices to large model input sizes is fundamentally solved. Traditional stretching and enlargement methods artificially increase pixels through interpolation algorithms, resulting in blurred edges of tiny debris only a few pixels wide, and even artifacts, severely interfering with the model's feature recognition. The fill strategy of this application, however, can completely preserve the original pixel resolution and texture details of the target in the slice image, ensuring that distant small targets (such as gravel or parts on the road surface) have the highest realism and fidelity when input into the model. This maximizes the visual language model's ability to perceive fine-grained features and improves the detection accuracy for extremely small targets.

[0080] In one embodiment, the process of constructing the training data for the pre-fine-tuned large visual language model includes:

[0081] Obtain the original training image containing road debris annotations, wherein the annotations are global coordinates based on the coordinate system of the original training image;

[0082] The original training image is divided into training slice images using the same strategy as in step S102;

[0083] The coordinates of the top-left vertex of the training slice image in the original training image are calculated as the offset.

[0084] Subtract the offset from the global coordinates of the road debris markers to convert them into local coordinates relative to the training slice image;

[0085] The large visual language model is fine-tuned using the training slice images and their corresponding local coordinate annotations.

[0086] Specifically, to ensure that the large visual language model can adapt to the slice inference mode during actual deployment, the system adopts a data construction strategy that is strictly aligned with online inference during the offline training phase. First, a large number of original training images containing road debris are acquired. These images have been manually labeled, and the coordinates of the labeled boxes are based on a global coordinate system with the top-left corner of the original image as the origin. Then, the system performs a crucial data preprocessing step: using a dynamic vertical partitioning and slicing strategy completely consistent with step S102, each high-resolution original training image is cropped into multiple training slice images. For each slice containing debris labels, the system calculates the coordinates of the top-left corner vertex of the slice in the original image as a position offset. Next, the system executes a coordinate transformation algorithm, subtracting the position offset from the global coordinates of the original labels to calculate the local coordinates relative to the top-left corner of the slice image. For example, if the original label is (x1, y1, x2, y2) and the top-left corner of the slice is (x0, y0), the transformed coordinates are (x1-x0, y1-y0, x2-x0, y2-y0). Finally, the system uses the cropped training slice images as input and the transformed local coordinates as supervision labels to construct a pairwise fine-tuning dataset, which is then input into the large visual language model for training, enabling the model to learn to identify and locate targets from the slice perspective.

[0087] In this embodiment, consistency between training and inference is achieved by strictly reusing the slicing strategy and coordinate transformation logic from the inference stage during the training data construction phase. This ensures that the image resolution, target pixel ratio, and background context range during model training are completely consistent with those in actual applications. This strategy eliminates biases in data distribution, enabling the model to more accurately adapt to local features after slicing, significantly improving the convergence speed and detection accuracy of the model in practical engineering applications.

[0088] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0089] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0090] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0091] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0092] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0093] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0094] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0095] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0096] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0098] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0099] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0101] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0102] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A method for detecting road spills based on roadside cameras, characterized in that, The method includes: Acquire real-time road video streams captured by roadside cameras and extract single-frame images; The single-frame image is divided into multiple detection regions, and a corresponding slice window is set for each detection region for sampling; wherein, the slice window size of the detection region located near the top of the single-frame image is smaller than the slice window size of the detection region located near the bottom of the single-frame image; The image sampled from the slice window is input into a pre-tuned large visual language model, and the target detection result is output. Based on the position coordinates of the slice window in the single-frame image, the target detection result is mapped back to the coordinate system of the single-frame image to obtain the suspected spill detection result of the single-frame image.

2. The method according to claim 1, characterized in that, After obtaining the suspected debris detection result of the single frame image, the method further includes performing a spatiotemporal consistency check on the suspected debris detection result, specifically including: Within a set time window, extract the detection results of the suspected spilled material from multiple consecutive frames of images; Calculate the positional offset of the suspected spill detection result between the consecutive multiple image frames; If the positional offset is less than a preset static threshold, it is identified as road debris. If the position offset is greater than the stationary threshold, it is identified as a moving interference target and removed from the detection results.

3. The method according to claim 1, characterized in that, The step of setting a corresponding slice window for sampling for each detection region specifically includes: Within the same detection area, adjacent slice windows have overlapping areas; The overlap rate between the slice windows is negatively correlated with the size of the slice windows.

4. The method according to claim 1, characterized in that, Prior to the step of dividing the single-frame image into multiple detection regions, the method further includes: The single-frame image is identified using a semantic segmentation model or a large visual language model to generate a road mask that only contains the road surface area. Multiple detection areas are divided only within the image area covered by the road mask.

5. The method according to claim 1, characterized in that, The step of dividing the single-frame image into multiple detection regions further includes: Identify the vanishing point or horizon position of the road lanes in the single-frame image; Based on the vanishing point or horizon position, a perspective depth gradient model is constructed along the vertical direction of the image; Based on the perspective depth gradient model, the ground sampling distance corresponding to pixels at different vertical coordinates in the image is calculated, and the boundary line of the multiple detection areas is determined according to the numerical distribution of the ground sampling distance.

6. The method according to claim 1, characterized in that, After the step of mapping the target detection result back to the coordinate system of the single-frame image to obtain the suspected debris detection result of the single-frame image, the method further includes: Perform pairwise traversal of all target detection boxes mapped back to the coordinate system of the single-frame image; Calculate the intersection-union ratio (CUP) between each pair of object detection boxes; If the cross-union ratio is greater than the preset fusion threshold, it is determined to be a duplicate detection, and the target detection box with the higher confidence score is retained, or the target detection boxes of the duplicate detection are weighted and averaged based on the confidence score to be merged into a single target detection box.

7. The method according to claim 1, characterized in that, The large visual language model is fine-tuned using a strategy that includes hard example mining: Construct a difficult example dataset that includes road traffic markings, tree shadows, and roadside facilities; The difficult example dataset is added to the training data to fine-tune the pre-trained visual language large model.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images