Road surface residual detection method, device, equipment and storage medium

Abstract

The application provides a road surface residual detection method, device and equipment and a storage medium, and relates to the technical field of data recognition. The method comprises the following steps: initializing a background library based on all instances included in the first number of previous video frames and the structured features corresponding to each instance; updating the background library based on the structured features of each instance in the current video frame and the baseline detection frame of the background instance in the background library; screening out the foreground instance in the current video frame based on the IoU combined with the IoF; and screening out the road surface residual based on the category label of the foreground instance and the preset whitelist. By adopting the IoU combined with the IoF, the application distinguishes whether all instances belong to the background instance or the foreground instance, and simultaneously updates the background library, so that the process of determining the foreground instance is more accurate, does not depend on data samples for training, improves the processing rate and the accuracy of calculation, and has wider applicability and can adapt to complex and changeable traffic scenes.

Description

Technical Field

[0001] This invention relates to the field of data recognition technology, and in particular to a method, apparatus, equipment and storage medium for detecting road debris. Background Technology

[0002] Highways experience heavy traffic and high speeds, making accidents caused by debris or litter on highways highly susceptible to serious consequences. These road obstructions pose a significant threat to the safety of drivers and passengers, increasing the likelihood of secondary accidents and severely impacting the overall traffic capacity and operational efficiency of highways. Therefore, real-time and accurate detection of debris or litter on highways is crucial for ensuring traffic safety.

[0003] Existing litter or debris detection algorithms can be divided into three main categories: traditional methods, supervised algorithms based on deep learning, and unsupervised algorithms based on deep learning. Traditional methods do not rely on data samples but have poor robustness; supervised methods based on deep learning outperform traditional methods but require a large number of data samples and have poor generalization ability. Unsupervised methods based on deep learning achieve a good balance between performance and practicality, but their performance is poor and the types of debris they can detect are limited. However, existing road litter detection methods based on deep learning supervised or unsupervised algorithms cannot handle unknown litter categories outside of the data samples used as training sets, making them difficult to adapt to complex and ever-changing traffic scenarios.

[0004] Therefore, reducing or eliminating the dependence on data samples in methods for detecting road debris or residues, thereby improving the accuracy of identifying road debris or residues, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention provides a method, apparatus, equipment, and storage medium for detecting road debris.

[0006] In a first aspect, the present invention provides a method for detecting road debris, comprising:

[0007] Based on all instances included in the first number of video frames, and the structured features corresponding to each instance, a background library is initialized; the structured features include instance identifiers, detection boxes, and category labels; the background library includes one or more baseline detection boxes corresponding to background instances; the first number is a positive integer;

[0008] The background library is updated based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library;

[0009] Based on the IoU combined with IoF method and the background library, foreground instances are filtered out from all instances in the current video frame;

[0010] Based on the category tags corresponding to the foreground instances and a preset whitelist, the foreground instances corresponding to road debris are filtered out.

[0011] Optionally, initializing the background library based on all instances included in the first number of video frames and the structured features corresponding to each instance includes:

[0012] Based on the instance identifiers of each instance in the first number of video frames and the detection boxes of each instance, determine the motion trajectory formed by all detection boxes of all instances belonging to the same instance identifier;

[0013] A baseline detection box corresponding to each background instance is determined based on the average value of the motion trajectories having the same instance identifier.

[0014] Based on the baseline detection box corresponding to the background instance and the instance identifier corresponding to the background instance, initialize the background library;

[0015] The first number of video frames include all instances, and the structured features corresponding to each instance are obtained based on an improved two-stage target detector model.

[0016] Optionally, updating the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library includes:

[0017] Determine the first IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0018] The background library is updated based on the first IoU value and the preset movement threshold.

[0019] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0020] Optionally, updating the background database based on the first IoU value and a preset movement threshold includes:

[0021] If the first IoU value is greater than or equal to a preset movement threshold, then the baseline detection box of the target background instance is updated based on the detection box of the target instance.

[0022] If the first IoU value is less than the preset movement threshold, then the target background instance is deleted from the background library.

[0023] Optionally, the method of filtering foreground instances from all instances in the current video frame based on IoU combined with IoF, and the background library, includes:

[0024] Determine the first IoF value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0025] Determine a second IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance;

[0026] Based on the relationship between the first IoF value and the preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, the target instance is determined to be either a foreground instance or a background instance.

[0027] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0028] Optionally, determining whether the target instance is a foreground instance or a background instance based on the relationship between the first IoF value and a preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, includes:

[0029] If both the first IoF value and the second IoU value are less than a preset foreground threshold, then the target instance is determined to be a foreground instance.

[0030] If the first IoF value or the second IoU value is greater than or equal to a preset foreground threshold, then the target instance is determined to be a background instance.

[0031] Optionally, the step of filtering out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist includes:

[0032] Foreground instances whose category tags are not in the preset whitelist are selected as foreground instances corresponding to road debris.

[0033] The preset whitelist includes category labels corresponding to all vehicle types in the traffic scenario.

[0034] Secondly, the present invention also provides a road surface debris detection device, comprising:

[0035] An initialization module is used to initialize a background library based on all instances included in the first number of video frames and the structured features corresponding to each instance; the structured features include instance identifiers, detection boxes, and category labels; the background library includes one or more baseline detection boxes corresponding to background instances; the first number is a positive integer;

[0036] The background instance module is used to update the background library based on the structured features of each instance in the current video frame and the baseline detection box corresponding to the background instance in the background library;

[0037] The foreground instance module is used to filter out foreground instances from all instances in the current video frame based on the IoU combined with IoF method and the background library;

[0038] The filtering module is used to filter out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist.

[0039] Thirdly, the present invention also provides an electronic device, including a memory, a transceiver, and a processor;

[0040] A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs in the memory and implementing the road surface debris detection method described in the first aspect above.

[0041] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon for causing a computer to perform the road debris detection method described in the first aspect above.

[0042] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the road debris detection method described in the first aspect above.

[0043] The road debris detection method, apparatus, equipment, and storage medium provided by this invention distinguish whether each instance belongs to the background or the foreground by using the IoU and IoF method to distinguish the instances included in each video frame and the structured features of each instance. At the same time, the initialized background library is updated, making the process of determining the foreground instance more accurate. It does not need to rely on data samples for training, improves the processing speed and calculation accuracy, has wider applicability, and can adapt to complex and ever-changing traffic scenarios. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the method for detecting road debris provided by the present invention;

[0046] Figure 2 This is a schematic diagram of the intersection-union ratio (CUIR) results provided by the present invention;

[0047] Figure 3 This is a schematic diagram of the foreground instance filtering process provided by the present invention;

[0048] Figure 4 This is a schematic diagram of the overall process of the road surface debris detection method provided by the present invention;

[0049] Figure 5 This is a schematic diagram of the road debris detection device provided by the present invention;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] The following is based on Figures 1 to 6 This invention describes the method, apparatus, equipment, and storage medium for detecting road debris provided by the present invention.

[0053] Figure 1 This is a flowchart illustrating the method for detecting road debris provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0054] Step 101: Based on all instances included in the first number of video frames and the structured features corresponding to each instance, initialize the background library; the structured features include instance identifiers, detection boxes, and category labels; the background library includes one or more baseline detection boxes corresponding to background instances; the first number is a positive integer;

[0055] Specifically, after obtaining all instances of the first number of video frames and the structured features corresponding to each instance, it is necessary to further determine whether each instance is moving or stationary, that is, whether each instance is a foreground instance or a background instance.

[0056] The first few video frames refer to the first few video frames from the start time. This first number can be set empirically and adjusted based on actual results, or any value can be set and adjusted based on actual results. For example, it could be randomly set to 100 or set to 300 based on previous statistical results. Based on all instances included in the first few video frames and the structured features corresponding to each instance, the detection boxes of all instances with the same instance identifier are averaged or weighted to form a baseline detection box for a background instance. The detection box may be an irregular border or a regular quadrilateral. It is sufficient to ensure that all pixel features of the instance are included in the detection box. For example, if the detection box is a quadrilateral, it can be represented by the coordinates of its four vertices. The coordinates of the detection boxes of all instances with the same instance identifier are averaged sequentially according to the order of the four vertices to obtain a new set of four vertex coordinates, which serves as the baseline detection box for the background instance represented by that instance identifier. Alternatively, the detection boxes farther away from the current time point have smaller weights, and the detection boxes closer to the current time point have larger weights. A new detection box is obtained by weighted summing of the detection boxes of all instances with the same instance identifier, and this new detection box serves as the baseline detection box for the background instance represented by that instance identifier. This baseline detection box for the background instance can be understood as a standard detection box for a background instance. Detection boxes of instances included in other frames can be compared with the baseline detection box of the background instance with the same instance identifier to further determine whether the instance is a background instance.

[0057] Using the same method, a baseline detection box representing the background instance for each instance identifier can be determined. The resulting background library can be specifically represented as follows: Where i, j, and k represent instance identifiers. This represents the baseline detection box for the i-th instance. This represents the baseline detection box for the j-th instance. This represents the baseline detection box for the k-th instance.

[0058] Step 102: Update the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library;

[0059] Specifically, the background library is initialized by taking all instances from the first few video frames and their corresponding structured features. This means that all instances in the initial background library are considered background instances. Based on the detection boxes of all instances with the same instance identifier in each video frame, an averaging or weighted summation method is used to determine the baseline detection box for any background instance. Then, the detection boxes of each instance in subsequent video frames are compared with the baseline detection boxes of the background instances. Based on the comparison result, it is determined whether the instance belongs to the background or not (belongs to the foreground). This comparison can be done by calculating the Intersection over Union (IoU) value or the degree of overlap. The initialized background library is then updated based on the result of whether the instance belongs to the background or foreground, making the background library data more consistent with the actual scene. This is because the initialization process is equivalent to assuming that all instances belong to the background, but this may not be the case in reality. The formula for the IoU can be expressed as:

[0060]

[0061] Where IoU represents the intersection-union ratio, A represents the detection box of any instance in the current video frame, and B represents the baseline detection box of the background instance that has the same instance identifier as the instance represented by A.

[0062] The larger the overlap area between A and B, the larger the corresponding IoU value; conversely, the smaller the overlap area between A and B, the smaller the corresponding IoU value. The IoU value ranges from 0 to 1. When the detection bounding box of a certain instance in the current video frame completely overlaps with the reference detection bounding box of a background instance with the same instance identifier, the corresponding IoU value is 1.

[0063] Figure 2 This is a schematic diagram of the intersection-union ratio (IU) results provided by the present invention, as shown below. Figure 2 As shown, box A represents the region corresponding to the detection box of any instance in the current video frame, box B represents the region corresponding to the reference detection box of the background instance that has the same instance identifier as the instance represented by A, the A∩B region filled with diagonal lines represents the region where the detection box of any instance in the current video frame and the reference detection box of the background instance that has the same instance identifier as the instance represented by A overlap, and the white areas of box A, box B and the A∪B region filled with diagonal lines represent the regions covered by the detection box of any instance in the current video frame and the reference detection box of the background instance that has the same instance identifier as the instance represented by A.

[0064] Step 103: Based on the IoU and IoF method and the background library, filter out the foreground instances in all instances of the current video frame;

[0065] After updating the background library using the method described above, a more accurate baseline detection box for each background instance is obtained. Based on this, an IoU (Intersection over Foreground) combined with the Intersection over Foreground (IoF) method is used to determine whether all instances in the current video frame are in the background library. If an instance is in the background library, the background library is updated; otherwise, the instance is considered a foreground instance. This IoU-IoF combination avoids the situation where the detection box of an instance might become smaller when the object is occluded. Directly relying on the IoU value could lead to misclassifying an instance that belongs to the background as a foreground instance. Therefore, after using IoU, combining it with IoF determines the IoF value between the detection box of the instance in the current video frame and the baseline detection box of the background instance, further reducing the possibility of misclassifying an instance as a background instance when the object is occluded.

[0066] The formula corresponding to the above foreground crossover ratio can be expressed as:

[0067]

[0068] Where IoF represents the foreground crossover ratio, A represents the detection box of any instance in the current video frame, and B represents the baseline detection box of the background instance that has the same instance identifier as the instance represented by A.

[0069] The larger the overlap area between A and B, the larger the corresponding IoF value; conversely, the smaller the overlap area between A and B, the smaller the corresponding IoF value. The IoF value ranges from 0 to 1. The IoU value between a detection box for any instance in the current video frame and a reference detection box for a background instance with the same instance identifier is less than the IoF value between the two detection boxes mentioned above.

[0070] Step 104: Based on the category label corresponding to the foreground instance and the preset whitelist, filter out the foreground instances corresponding to road debris.

[0071] After identifying the foreground instances among all instances in the current video frame, the system further determines whether the category labels of these foreground instances are in a preset whitelist. If they are in the whitelist, the foreground instance is classified as a non-road debris. If not, it is classified as a road debris. This preset whitelist primarily defines common moving instances (foreground instances) in traffic scenes, such as different types of cars and pedestrians.

[0072] The road debris detection method provided by this invention distinguishes between background and foreground instances by using IoU and IoF methods to identify instances in each video frame and their structured features. It also updates the initialized background database, making the process of determining foreground instances more accurate. This method does not require training based on data samples, improves processing speed and computational accuracy, has wider applicability, and can adapt to complex and ever-changing traffic scenarios.

[0073] Optionally, initializing the background library based on all instances included in the first number of video frames and the structured features corresponding to each instance includes:

[0074] Based on the improved two-stage target detector model, all instances included in the first number of video frames are obtained, as well as the structured features corresponding to each instance.

[0075] Based on the instance identifiers of each instance in the first number of video frames and the detection boxes of each instance, determine the motion trajectory formed by all detection boxes of all instances belonging to the same instance identifier;

[0076] A baseline detection box corresponding to each background instance is determined based on the average value of the motion trajectories having the same instance identifier.

[0077] The background library is initialized based on the baseline detection box corresponding to the background instance and the instance identifier corresponding to the background instance.

[0078] Specifically, monitoring videos of the road surface are acquired. All video frames are identified using the improved two-stage object detector model provided by this invention, which identifies all instances included in each video frame and the corresponding structured features of each instance. During background library initialization, the first few video frames from the monitoring video are extracted. Then, frame by frame, the improved two-stage object detector model is used to identify all instances included in each video frame and the corresponding structured features of each instance. The improved two-stage object detector model introduces a Contrastive Language Image Pre-training (CLIP) model as a classification head to achieve zero-shot object detection based on the two-stage object detector model. This detector uses image-level training data to assist in object detection training through a specific classification loss function, thereby achieving stronger zero-shot detection capabilities. The two-stage object detector model is built based on Faster Region-based Convolutional Neural Networks (Faster RCNN), a commonly used object detection method, and consists of two parts: candidate location extraction and secondary discrimination of candidate locations. Compared to the text-image pre-trained model CLIP, which is self-supervised based on massive amounts of internet image and text data, CLIP can perform image and open-vocabulary text matching after training. Simply put, it only requires a text description of the image category to classify the image. The specific classification loss function is a specially designed function, characterized by introducing features from the image-text pre-trained model to guide the classifier's parameter learning on top of traditional object detection methods. Based on this, it determines whether instances in different video frames belong to the same object to be detected, mainly through Kalman filtering and the Hungarian algorithm for position matching. It obtains all instances in any adjacent video frames and their corresponding bounding boxes. The maximum overlap between any two bounding boxes indicates that the instances corresponding to these two bounding boxes belong to the same target, i.e., the same object to be detected.

[0079] An improved two-stage object detector model identifies all instances in each video frame and the corresponding structured features of each instance. The structured features of each instance specifically include an instance identifier, a bounding box, and a category label. The instance identifier uniquely identifies any object to be detected. In different video frames, the same object may correspond to multiple instances, but these instances share the same instance identifier, and each instance has a corresponding bounding box and category label. (Any video frame I) t After using the improved two-stage target detector model, a set consisting of all instances is obtained. This represents the i-th instance in the t-th video frame, and its corresponding structured features include the detection box. and category labels Let represent the detection bounding box of the i-th instance in the t-th video frame. Let represent the category label of the i-th instance in the t-th video frame, where i represents the instance identifier. All instances can be divided into two categories: foreground instances and background instances. Background instances can be simply understood as static instances, and foreground instances can be simply understood as moving instances.

[0080] Based on the improved two-stage target detector model, after acquiring all instances included in all video frames and the structured features corresponding to each instance, the first number of video frames from the start time are extracted. Based on all instances in the first number of video frames and the structured features corresponding to each instance, the detection boxes of all instances with the same instance identifier are determined as the motion trajectory of the background instance represented by the instance identifier. This can be understood as the same instance identifier having a detection box corresponding to the instance in multiple video frames. These detection boxes are averaged to determine a new detection box, which is used as the reference detection box of the background instance represented by the instance identifier.

[0081] The detection boxes of multiple instances belonging to the same instance identifier are averaged. Specifically, the maximum and minimum values ​​of the corresponding positions of each boundary point of each detection box are determined, the average of the maximum and minimum values ​​is taken, and the position information of each boundary point is updated to obtain a new detection box, which serves as the reference detection box for the background instance represented by the instance identifier.

[0082] After determining the baseline detection box of the background instance represented by each instance identifier according to the above method, the initialization of the background library is completed. The background library includes baseline detection boxes corresponding to one or more background instances.

[0083] Optionally, updating the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library includes:

[0084] Determine the first IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0085] The background library is updated based on the first IoU value and the preset movement threshold.

[0086] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0087] Specifically, after initializing the background library, it is necessary to update the background library in real time or periodically by combining the detection boxes of each instance in the new video frame.

[0088] Obtain any instance from all instances in the current video frame as the target instance, along with the target instance's structured features, such as instance identifier, bounding box, and category label. Identify a background instance in the background library that shares the same instance identifier as the target instance, and use it as the target background instance, along with its corresponding baseline bounding box.

[0089] The Intersection over Union (IoU) method is used to determine the IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance, i.e., the first IoU value. This first IoU value is compared with a preset motion threshold to determine whether the target instance is a background instance. If it is a background instance, the background library information needs to be updated. This might involve updating the reference detection bounding box corresponding to an existing background instance, deleting a background instance, or adding a background instance. The preset motion threshold is mainly used to distinguish whether instances with the same instance identifier in different video frames have moved. It can be set according to actual needs. If the preset motion threshold is too small, instances with shaking might be judged as having moved; if the preset motion threshold is set too large, instances that have actually moved might be mistakenly judged as not having moved. Therefore, the preset motion threshold needs to be set reasonably according to the actual scene and actual needs, and can also be adjusted based on actual results in subsequent processes. The formula for the above Intersection over Union can also be expressed as:

[0090]

[0091] Where IoU represents the intersection-union ratio, This represents the detection box of the i-th instance in the t-th video frame. Let t represent the baseline detection box of the i-th background instance, where t is any positive integer and i is the instance identifier. Instances with the same instance identifier belong to the same detection object.

[0092] Optionally, updating the background database based on the first IoU value and a preset movement threshold includes:

[0093] If the first IoU value is greater than or equal to a preset movement threshold, then the baseline detection box of the target background instance is updated based on the detection box of the target instance.

[0094] If the first IoU value is less than the preset movement threshold, then the target background instance is deleted from the background library.

[0095] Specifically, the IoU method is used to determine the IoU value between the detection box of the target instance and the reference detection box of the target background instance, i.e., the first IoU value. Then, the relationship between the first IoU value and the preset movement threshold is compared to update the initialized background library.

[0096] If the first IoU value is greater than or equal to the preset movement threshold, it indicates that the overlap between the target instance and the target background instance is high. In this case, the target instance is determined to be a background instance. The average value of the detection box of the target instance and the baseline detection box of the target background instance is calculated again, and the baseline detection box of the target background instance is updated.

[0097] If the first IoU value is less than the preset movement threshold, it indicates that the overlap between the target instance and the target background instance is low. Therefore, it is determined that the target instance may have moved, and thus it is determined that the target background instance with the same instance identifier does not belong to the background instance. The target background instance is then deleted from the background library.

[0098] The above method allows for real-time or periodic updates of the background library, depending on application requirements. For periodic updates, the period value can be set to the duration of the first few video frames or any other duration.

[0099] Optionally, the method of filtering foreground instances from all instances in the current video frame based on IoU combined with IoF, and the background library, includes:

[0100] Determine the first IoF value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0101] Determine a second IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance;

[0102] Based on the relationship between the first IoF value and the preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, the target instance is determined to be either a foreground instance or a background instance.

[0103] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0104] Specifically, after updating the background library, any instance in the current video frame is selected as the target instance. A background instance with the same instance identifier as the target instance is then retrieved from the background library; this is the target background instance. Combining IoU and IoF (Intersection over Union) values, the IoU value between the detection box of the target instance and the target background instance, as well as the IoF value, are determined. The IoU value and the IoF value are then compared with a preset foreground threshold to determine whether the target instance is a foreground instance or a background instance. The preset foreground threshold can be set according to requirements or based on previous statistical results, and can be continuously adjusted during subsequent implementation.

[0105] In instance detection, objects are sometimes occluded. When an object is occluded, the improved two-stage object detector model identifies the object and obtains the corresponding instance and its structured features. However, the bounding box of this instance may become smaller due to the occlusion. If the IoU-calculated metric is still used for evaluation, the visible portion of the object may be misclassified as not belonging to the same instance identified in previous video frames. To balance the sensitivity and robustness to object occlusion, a method combining IoU and IoF is proposed, using IoF as a supplement to IoU. The formula for the foreground crossover ratio can also be expressed as:

[0106]

[0107] Where IoF represents the foreground crossover ratio, This represents the detection box of the i-th instance in the t-th video frame. Let t represent the baseline detection box for the i-th background instance, where t is any positive integer and i is the instance identifier. Instances with the same instance identifier belong to the same detection object. For example... Figure 2 As shown, it is represented by box A. The corresponding area is represented by box B. The corresponding area, represented by the diagonally filled A∩B region. The corresponding region. The larger the overlap between the detection box of the i-th instance and the reference detection box of the i-th background instance in the current video frame, the larger the corresponding IoF value; conversely, the smaller the overlap, the smaller the corresponding IoF value. The IoF value ranges from 0 to 1. When the detection box of the i-th instance and the reference detection box of the i-th background instance in the current video frame completely overlap, the corresponding IoF value is 1. The IoU value between the detection box of any instance in the current video frame and the reference detection box of a background instance with the same instance identifier is less than the IoF value between the two detection boxes mentioned above.

[0108] In the foreground instance selection stage, the detection results of the current video frame are matched with background instances in the background library. An IoU (Inversion of Union) combined with IoF (Inversion of Frame) approach is used. This involves determining the detection bounding boxes of instances with the same instance identifier and the baseline detection bounding box of the background instance, and determining the IoU and IoF values ​​between them. Based on these IoU and IoF values, it is determined whether the instance in the current video frame belongs to the foreground. The specific process can be represented as follows: Figure 3 As shown:

[0109] Retrieve all instances and the background library included in the t-th video frame;

[0110] Extract the detection bounding boxes of instance i with the same instance identifier i from all instances included in the t-th video frame and from the background library. Benchmark detection boxes for background instances

[0111] Determine the detection box based on the combination of IoF and IoU. and reference detection frame The IoU and IoF values, for the same instance identifier i, are used to determine the bounding box. and reference detection frame The corresponding first IoF value is usually greater than the detection box of instance identifier i. and reference detection frame The corresponding second IoU value;

[0112] Compare the above IoU value with the preset foreground threshold, the above IoF value with the preset foreground threshold, that is, compare the second IoU value with the preset foreground threshold, and the first IoF value with the preset foreground threshold;

[0113] Based on the above size relationship, we can further determine whether instance i is a foreground instance.

[0114] Optionally, determining whether the target instance is a foreground instance or a background instance based on the relationship between the first IoF value and a preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, includes:

[0115] If both the first IoF value and the second IoU value are less than a preset foreground threshold, then the target instance is determined to be a foreground instance.

[0116] If the first IoF value or the second IoU value is greater than or equal to a preset foreground threshold, then the target instance is determined to be a background instance.

[0117] Specifically, if both the first IoF value and the second IoU value are less than a preset foreground threshold, then the instance is considered to have moved and is determined to be a foreground instance. Conversely, if either the first IoF value or the second IoU value is greater than or equal to the preset foreground threshold, then the instance is considered not to have moved and is determined to be a foreground instance.

[0118] The corresponding selection rules mentioned above can be expressed as:

[0119]

[0120] in, Let FG represent the i-th instance in the t-th video frame, BG represent the foreground instance, and T represent the background instance. fore Indicates the preset foreground threshold. This represents the detection box of the i-th instance in the t-th video frame. This represents the baseline detection box for the i-th background instance, where i is the instance identifier. express and The IoU value between them express and The IoF value between them.

[0121] Optionally, the step of filtering out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist includes:

[0122] Foreground instances whose category tags are not in the preset whitelist are selected as foreground instances corresponding to road debris.

[0123] The preset whitelist includes category labels corresponding to all vehicle types in the traffic scenario.

[0124] Specifically, after filtering out foreground instances in the current video frame, these foreground instances, although in motion, are not necessarily all road debris. Further determination of whether these foreground instances are road debris is needed based on a preset whitelist. This preset whitelist typically consists of category labels for all vehicles in a traffic scene, such as pedestrians, buses, and trucks. After filtering out foreground instances with these category labels, the remaining foreground instances are considered road debris.

[0125] The road debris detection method provided by this invention distinguishes between background and foreground instances by using IoU and IoF methods to identify instances in each video frame and their structured features. It also updates the initialized background database, making the process of determining foreground instances more accurate. This method does not require training based on data samples, improves processing speed and computational accuracy, has wider applicability, and can adapt to complex and ever-changing traffic scenarios.

[0126] Figure 4 This is a schematic diagram of the overall process of the road surface debris detection method provided by the present invention, as shown below. Figure 4 As shown, the method includes:

[0127] Step 401: Obtain the first N video frames of the entire video file and perform detection using the full instance detection model. The full instance detection model here is an improved two-stage object detector model, which introduces the CLIP (Contrast Text Image Pre-trained Model) as the classification head for zero-shot object detection based on the two-stage object detector model. N is a positive integer, and the specific value is set according to requirements.

[0128] Step 402: After the full instance detection model has completed the detection of the first N video frames, the instance of each video frame and the structured features of each instance are obtained.

[0129] Step 403: Based on all instances in step 402 and the structured features of each instance, initialize the background model (background library);

[0130] Step 404: Obtain the current video frame and input it into the full instance detection model for detection;

[0131] Step 405: After the full instance detection model has completed the detection of the current video frame, the instances of the current video frame and the structured features of each instance are obtained.

[0132] Step 406: Based on all instances in the current video frame and the structured features of each instance obtained in Step 405, and combined with the IoU method, determine whether each instance in the current video frame is a background instance in the initial background model (background library). If so, update the background model; if not, delete the background instance in the background library that has the same instance identifier as the instance. The background model update process is continuous. Once there is a new video frame, the instances detected by the full instance detection model are updated in the background library according to the above method.

[0133] Step 407: Obtain any instance in the current video frame. For example, based on instance identifier i, obtain the detection box corresponding to instance i, and at the same time, obtain the baseline detection box of the background instance with instance identifier i in the background library.

[0134] Step 408: Based on the foreground instance filtering rules using the IoU and IoF method, determine the IoU value and IoF value of the detection box corresponding to instance i and the baseline detection box of the background instance, and compare these two values ​​with the size of the preset foreground threshold included in the filtering rules.

[0135] Step 409: Filter out foreground instances. If both the IoU value and IoF value are less than the preset foreground threshold, then instance i is determined to be a foreground instance.

[0136] Step 410: Remove foreground instances corresponding to common category labels in traffic scenarios. Compare whether the category label in the structured features of instance i is among the common category labels in traffic scenarios. If it is, then determine that the instance is not a road debris; if not, then determine that the instance is a road debris.

[0137] Step 411: Output the road debris detection results. The user can be notified of the presence of road debris through alarms on the display screen or audible prompts.

[0138] Figure 5 This is a schematic diagram of the road debris detection device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0139] An initialization module 501 is used to initialize a background library based on all instances included in the first number of video frames and the structured features corresponding to each instance; the background library includes multiple background instances and baseline detection boxes of the background instances; the structured features include instance identifiers, detection boxes, and category labels; the baseline detection boxes of the background instances are the average of all detection boxes of each instance with the same instance identifier in the first number of video frames; the first number is a positive integer;

[0140] Background instance module 502 is used to update the background library based on the structured features of each instance in the current video frame and the baseline detection box corresponding to the background instance in the background library;

[0141] The foreground instance module 503 is used to filter out foreground instances from all instances in the current video frame based on the IoU combined with IoF method and the background library;

[0142] The filtering module 504 is used to filter out the foreground instances corresponding to road debris based on the category labels corresponding to the foreground instances and a preset whitelist.

[0143] Optionally, during the initialization module 501's process of initializing the background library based on all instances included in the first number of video frames and the structured features corresponding to each instance, it is specifically used for:

[0144] Based on the instance identifiers of each instance in the first number of video frames and the detection boxes of each instance, determine the motion trajectory formed by all detection boxes of all instances belonging to the same instance identifier;

[0145] A baseline detection box corresponding to each background instance is determined based on the average value of the motion trajectories having the same instance identifier.

[0146] Based on the baseline detection box corresponding to the background instance and the instance identifier corresponding to the background instance, initialize the background library;

[0147] The first number of video frames include all instances, and the structured features corresponding to each instance are obtained based on an improved two-stage target detector model.

[0148] Optionally, the background instance module 502, in updating the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library, specifically performs the following:

[0149] Determine the first IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0150] The background library is updated based on the first IoU value and the preset movement threshold.

[0151] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0152] Optionally, updating the background database based on the first IoU value and a preset movement threshold includes:

[0153] If the first IoU value is greater than or equal to a preset movement threshold, then the baseline detection box of the target background instance is updated based on the detection box of the target instance.

[0154] If the first IoU value is less than the preset movement threshold, then the target background instance is deleted from the background library.

[0155] Optionally, in the process of filtering foreground instances from all instances in the current video frame based on the IoU combined with IoF and the background library, the foreground instance module 503 is specifically used for:

[0156] Determine the first IoF value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0157] Determine a second IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance;

[0158] Based on the relationship between the first IoF value and the preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, the target instance is determined to be either a foreground instance or a background instance.

[0159] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0160] Optionally, determining whether the target instance is a foreground instance or a background instance based on the relationship between the first IoF value and a preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, includes:

[0161] If both the first IoF value and the second IoU value are less than a preset foreground threshold, then the target instance is determined to be a foreground instance.

[0162] If the first IoF value or the second IoU value is greater than or equal to a preset foreground threshold, then the target instance is determined to be a background instance.

[0163] Optionally, in the process of filtering out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist, the filtering module 504 is specifically used for:

[0164] Foreground instances whose category tags are not in the preset whitelist are selected as foreground instances corresponding to road debris.

[0165] The preset whitelist includes category labels corresponding to all vehicle types in the traffic scenario.

[0166] Specifically, the road debris detection device provided by the present invention can realize all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0167] Figure 6 This is a schematic diagram of the structure of an electronic device provided by the present invention; as shown. Figure 6As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a road debris detection method, such as the method including:

[0168] Based on all instances included in the first number of video frames, and the structured features corresponding to each instance, a background library is initialized; the structured features include instance identifiers, detection boxes, and category labels; the background library includes one or more baseline detection boxes corresponding to background instances; the first number is a positive integer;

[0169] The background library is updated based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library;

[0170] Based on the IoU combined with IoF method and the background library, foreground instances are filtered out from all instances in the current video frame;

[0171] Based on the category tags corresponding to the foreground instances and a preset whitelist, the foreground instances corresponding to road debris are filtered out.

[0172] Optionally, initializing the background library based on all instances included in the first number of video frames and the structured features corresponding to each instance includes:

[0173] Based on the instance identifiers of each instance in the first number of video frames and the detection boxes of each instance, determine the motion trajectory formed by all detection boxes of all instances belonging to the same instance identifier;

[0174] A baseline detection box corresponding to each background instance is determined based on the average value of the motion trajectories having the same instance identifier.

[0175] Based on the baseline detection box corresponding to the background instance and the instance identifier corresponding to the background instance, initialize the background library;

[0176] The first number of video frames include all instances, and the structured features corresponding to each instance are obtained based on an improved two-stage target detector model.

[0177] Optionally, updating the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library includes:

[0178] Determine the first IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0179] The background library is updated based on the first IoU value and the preset movement threshold.

[0180] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0181] Optionally, updating the background database based on the first IoU value and a preset movement threshold includes:

[0182] If the first IoU value is greater than or equal to a preset movement threshold, then the baseline detection box of the target background instance is updated based on the detection box of the target instance.

[0183] If the first IoU value is less than the preset movement threshold, then the target background instance is deleted from the background library.

[0184] Optionally, the method of filtering foreground instances from all instances in the current video frame based on IoU combined with IoF, and the background library, includes:

[0185] Determine the first IoF value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame;

[0186] Determine a second IoU value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance;

[0187] Based on the relationship between the first IoF value and the preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, the target instance is determined to be either a foreground instance or a background instance.

[0188] The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

[0189] Optionally, determining whether the target instance is a foreground instance or a background instance based on the relationship between the first IoF value and a preset foreground threshold, and the relationship between the second IoU value and the preset foreground threshold, includes:

[0190] If both the first IoF value and the second IoU value are less than a preset foreground threshold, then the target instance is determined to be a foreground instance.

[0191] If the first IoF value or the second IoU value is greater than or equal to a preset foreground threshold, then the target instance is determined to be a background instance.

[0192] Optionally, the step of filtering out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist includes:

[0193] Foreground instances whose category tags are not in the preset whitelist are selected as foreground instances corresponding to road debris.

[0194] The preset whitelist includes category labels corresponding to all vehicle types in the traffic scenario.

[0195] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] It should be noted that the electronic device provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0197] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the road residue detection method provided in the above embodiments.

[0198] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program for causing a computer to execute the road debris detection method provided in the above embodiments.

[0199] On the other hand, the present invention also provides a processor-readable storage medium storing a computer program for causing the processor to execute the road debris detection method provided in the above embodiments.

[0200] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0201] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting road debris, characterized in that, include: The background library is initialized based on all instances included in the first number of video frames and the structured features corresponding to each instance. The structured features include instance identifiers, bounding boxes, and category labels; the background library includes one or more baseline bounding boxes corresponding to background instances; the first number is a positive integer; The background library is updated based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library; Based on the method of combining the Intersection over Union (IoU) ratio with the Foreground Crossover (IoF) ratio, and the background library, foreground instances are selected from all instances in the current video frame; Based on the category tags corresponding to the foreground instances and a preset whitelist, the foreground instances corresponding to road debris are filtered out. The method based on Intersection over Union (IoU) combined with Foreground Crossover Ratio (IoF), and the background library, filters out foreground instances from all instances in the current video frame, including: Determine the first foreground cross ratio (IoF) value between the detection bounding box of the target instance and the baseline detection bounding box of the target background instance in the current video frame; Determine the second intersection-union ratio (IoU) value between the detection bounding box of the target instance and the reference detection bounding box of the target background instance; If both the first foreground crossover ratio (IoF) and the second crossover-union ratio (IoU) are less than a preset foreground threshold, then the target instance is determined to be a foreground instance. If the first foreground crossover ratio (IoF) or the second crossover union ratio (IoU) is greater than or equal to a preset foreground threshold, then the target instance is determined to be a background instance. The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

2. The method for detecting road debris according to claim 1, characterized in that, The initialization of the background library based on all instances included in the first number of video frames and the structured features corresponding to each instance includes: Based on the instance identifiers of each instance in the first number of video frames and the detection boxes of each instance, determine the motion trajectory formed by all detection boxes of all instances belonging to the same instance identifier; A baseline detection box corresponding to each background instance is determined based on the average value of the motion trajectories having the same instance identifier. Based on the baseline detection box corresponding to the background instance and the instance identifier corresponding to the background instance, initialize the background library; The first number of video frames include all instances, and the structured features corresponding to each instance are obtained based on an improved two-stage target detector model.

3. The method for detecting road debris according to claim 1, characterized in that, The step of updating the background library based on the structured features of each instance in the current video frame and the baseline detection boxes corresponding to the background instances in the background library includes: Determine the first intersection-union ratio (IoU) between the detection bounding box of the target instance and the reference detection bounding box of the target background instance in the current video frame; The background library is updated based on the first Intersection over Union (IoU) value and a preset movement threshold. The target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance.

4. The method for detecting road debris according to claim 3, characterized in that, The step of updating the background library based on the first Intersection over Union (IoU) value and a preset movement threshold includes: If the first Intersection over Union (IoU) value is greater than or equal to a preset movement threshold, then the baseline detection box of the target background instance is updated based on the detection box of the target instance. If the first Intersection over Union (IoU) value is less than the preset movement threshold, then the target background instance is deleted from the background library.

5. The method for detecting road debris according to claim 1, characterized in that, The step of filtering out foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist includes: Foreground instances whose category tags are not in the preset whitelist are selected as foreground instances corresponding to road debris. The preset whitelist includes category labels corresponding to all vehicle types in the traffic scenario.

6. A road surface debris detection device, characterized in that, The device includes: An initialization module is used to initialize a background library based on all instances included in the first number of video frames and the structured features corresponding to each instance; the structured features include instance identifiers, detection boxes, and category labels; the background library includes one or more baseline detection boxes corresponding to background instances; the first number is a positive integer; The background instance module is used to update the background library based on the structured features of each instance in the current video frame and the baseline detection box corresponding to the background instance in the background library; A foreground instance module is used to filter foreground instances from all instances in the current video frame based on a combination of Intersection over Union (IoU) and Foreground Cross-Frequency (IoF), and the background library. The filtering of foreground instances from all instances in the current video frame based on IoU and Foreground Cross-Frequency, and the background library, includes: determining a first foreground cross-Frequency (IoF) value between the detection box of the target instance and the reference detection box of the target background instance in the current video frame; determining a second IoU value between the detection box of the target instance and the reference detection box of the target background instance; if both the first IoF value and the second IoU value are less than a preset foreground threshold, then the target instance is determined to be a foreground instance; if either the first IoF value or the second IoU value is greater than or equal to the preset foreground threshold, then the target instance is determined to be a background instance; the target instance is any one of all instances in the current video frame; the target background instance is a background instance in the background library that has the same instance identifier as the target instance. The filtering module is used to filter out the foreground instances corresponding to road debris based on the category tags corresponding to the foreground instances and a preset whitelist.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the road surface debris detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that causes a computer to perform the road debris detection method as described in any one of claims 1 to 5.

Images