A Highway Spill Detection and Classification Method Based on Open Vocabulary Detection
By constructing an open vocabulary list and combining it with SAM and CLIP models, high-precision detection and classification of debris on highways were achieved, solving the problems of difficulty in detecting small debris and low classification accuracy, and improving traffic safety.
Patent Information
- Application Number
- CN202410349437.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-26
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-26
AI Technical Summary
Existing technologies struggle with the detection of small and medium-sized debris, have low accuracy rates, rely on manual classification, require large amounts of training data, and lack generalization and transfer learning capabilities.
An open vocabulary list is constructed, and the Segment Anything Model (SAM) and Contrastive Language-Image Pre-training (CLIP) models are used to accurately extract and classify projectile targets by matching visual embeddings with text embeddings, thereby reducing the reliance on manually labeled data.
It improves the accuracy of spill detection and classification, exhibits better robustness in extreme environments, reduces the data requirements, and has better generalization performance and transfer learning capabilities, thus assisting in traffic safety early warning.
Smart Images

Figure CN118196726B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and intelligent transportation technology, and in particular to a method for detecting and classifying highway debris based on open vocabulary detection. Background Technology
[0002] In recent years, debris spillage on highways has become one of the main causes of major highway traffic accidents. Accurately detecting and classifying spilled debris is crucial for timely warnings and road cleanup, and is of great significance for ensuring highway traffic safety. However, due to the diversity of types, sizes, and locations of spilled debris, and the limitations imposed by the angle and position of highway cameras, the detection and classification of small spilled debris is particularly difficult. Especially in nighttime scenarios, poor lighting conditions and severe glare make it even more difficult to identify the shape, color, and other characteristics of the spilled debris.
[0003] Most current methods for detecting projectiles, such as semantic segmentation, frame difference, and background modeling, primarily focus on detecting the projectiles themselves while neglecting their type and properties. This offers little help in timely warnings and results in low accuracy and poor detection performance. While some deep learning methods can perform classification, the types of projectiles they can distinguish are limited by initial human-defined categories, and they require extensive training with manually labeled data to achieve relatively high accuracy. These methods lack adaptability to small projectiles and extreme environments, heavily rely on manually labeled data for training, struggle to accurately classify projectiles, and exhibit poor generalization and transfer learning capabilities. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a highway spill detection and classification method based on open vocabulary detection. This method solves the problems of existing technologies, such as difficulty in detecting small spills with low accuracy, reliance on limited human definitions for spill classification, and large requirements for training sets.
[0005] The objective of this invention can be achieved through the following technical solutions:
[0006] This invention provides a method for detecting and classifying highway spills based on open vocabulary detection, comprising the following steps:
[0007] S1: Construct an open vocabulary list for highway spills based on common spill types;
[0008] S2: The image is segmented. Based on the calculated mask size, the potential target regions of the projectiles are determined and obtained. During the extraction of the potential projectile target regions, the image containing the projectiles is processed based on the Segment Anything Model (SAM) to extract all possible target regions containing the projectiles.
[0009] S3: By using a contrastive language-image pre-trained model, the visual embedding of the potential target region is matched with the text embedding of the open vocabulary list. By calculating the confidence index, the projectile target is accurately extracted from the potential target region. Then, by calculating the similarity index, each extracted projectile is classified.
[0010] Furthermore, in S1, the specific process includes:
[0011] a1. Extract keywords related to spilled materials from highway surveillance videos and publicly reported data;
[0012] a2. Based on the nature of the spilled material, classify the keywords and add synonyms and near-synonyms for each keyword;
[0013] a3. Construct a network of relationships between each keyword and other words to build an open vocabulary list for highway spills.
[0014] Furthermore, in a1, the keywords include the name, color, shape, material, and state of the spilled material.
[0015] Furthermore, in a3, the relational network includes hierarchical relationships, correlations, and antonyms, and uses the color, shape, material, and scale features of the spilled object to establish a semantic relational network between words;
[0016] In A3, the list of open vocabulary is built from open-source data on the internet and new categories generated during testing. The initial categories include: garbage, goods, and dust.
[0017] Furthermore, in S2, the specific process includes:
[0018] b1. Initialize the SAM model, determine the number of sampling points and the minimum mask area based on the image pixels and sharpness, and automatically generate a mask for images containing projectiles;
[0019] b2. Calculate the IoU value. Based on the size of the projectile, the maximum mask area is individually divided. The mask area is calculated sequentially. Areas smaller than the maximum mask area are set as potential target areas. The bounding boxes and coordinates of these areas are recorded, and then the IoU value is calculated.
[0020] b3. Initially screen potential target regions. Use the non-maximum suppression algorithm to filter the potential target regions generated in b2, and select regions with IoU values higher than the preset threshold.
[0021] Furthermore, in b2, the formula for calculating the IoU value is as follows:
[0022]
[0023] in:
[0024] The union is the area of the predicted bounding box compared to the actual bounding box.
[0025] intersection is the area of the part where the two frames intersect.
[0026] Furthermore, in S3, the specific process includes:
[0027] c1. Projectile target recognition and localization: The image encoder of the Contrastive Language-Image Pre-training (CLIP) model is called to convert the latent target into a visual embedding, and the text encoder of the CLIP model is called to convert the open vocabulary list into a text embedding. The visual embedding of the latent target region and the text embedding of the open vocabulary list are matched through the contrastive language-image pre-training model.
[0028] c2. Combine the region bounding box and coordinates to calculate the confidence score and cosine similarity, and filter out the target region, i.e. the spilled area, from the potential target region. At the same time, add the newly detected spilled category to the open vocabulary list constructed in S1.
[0029] c3. Classification of spillage on highways: Match the target region obtained in step c2 to the open vocabulary list, calculate the probability of belonging to the category, and quantitatively analyze the semantic network between the most likely categories to determine the spillage category corresponding to the target region.
[0030] Furthermore, in c3, the top ten most likely categories of spilled material are selected as a reference. By analyzing the semantic network behind the selected categories, the categories and properties of the spilled material are extracted, ensuring the accuracy and reliability of the auxiliary information.
[0031] Furthermore, in S2 and S3, pre-trained models are used directly, with the SAM model using automatic search and the CLIP model using the ViT-B / 32 model.
[0032] In S2 and S3, the image data used includes both daytime and nighttime scenes.
[0033] Furthermore, it also includes S4: generating early warning information for road sections where debris is spilled, and feeding back the early warning information, the location and type of the spilled debris to the road monitoring terminal.
[0034] Compared with the prior art, the present invention has the following technical advantages:
[0035] 1) This invention proposes a highway debris detection and classification method based on open vocabulary detection. By generating potential target regions, the detection scope can be shifted from the entire image to regions, significantly improving accuracy. Compared with other detection methods, this method does not require training with a large amount of manually labeled data, and can also detect and identify new debris categories, exhibiting better generalization performance and transfer learning ability.
[0036] 2) For small, scattered objects, this method demonstrates better accuracy and higher recognition efficiency in extreme scenarios, and exhibits good robustness.
[0037] 3) In practical applications, this method helps to provide early warning of accidents and improve the overall traffic safety performance of highway operations. Attached Figure Description
[0038] Figure 1 This is a general flowchart of an embodiment of the present invention.
[0039] Figure 2 This is a technical roadmap for open vocabulary target detection in this embodiment of the invention.
[0040] Figure 3 This is a schematic diagram illustrating various types of spillage in a highway scenario, as described in an embodiment of the present invention.
[0041] Figure 4 This is a diagram showing the results of potential target region extraction based on SAM in an embodiment of the present invention.
[0042] Figure 5 This is a partial result image of the top ten target classification similarity rankings based on CLIP in an embodiment of the present invention. Detailed Implementation
[0043] Overall, this invention discloses a method for detecting and classifying highway debris based on open vocabulary detection, relating to the fields of computer vision and intelligent transportation technology. The method mainly includes the following steps: S1, constructing an open vocabulary list for highway debris, covering all possible debris categories, and updating it in real time based on road detection conditions; S2, segmenting the image, and determining and obtaining potential target regions of debris based on the calculated mask size; S3, accurately extracting debris targets from the potential target regions by calculating a confidence index, and classifying each extracted debris by calculating a similarity index; S4, issuing warnings for road sections where debris occurs, and feeding back information such as the location and category of the debris to the road monitoring terminal to take timely and appropriate measures to reduce traffic accidents. Compared to existing highway debris detection methods, this invention significantly reduces the data requirements and has better generalization ability and robustness.
[0044] The highway spill detection and classification method based on open vocabulary detection in this invention includes the following steps:
[0045] Step 1: Based on common types of spillage, construct an open vocabulary list for highway spillage. The specific steps are as follows:
[0046] a1. First, extract keywords related to the spilled material from data such as highway surveillance videos and related reports. The keywords mainly include the name, color, shape, material, and state of the spilled material.
[0047] a2. Based on the nature of the spilled material, classify the keywords and add synonyms and near-synonyms for each keyword;
[0048] a3. Construct a relationship network between each keyword and other words, including hierarchical relationships, relevance, antonyms, etc., and use the characteristics of the spilled object such as color, shape, material, and scale to establish a semantic relationship network between words.
[0049] Step 2: Potential Spray Object Target Region Extraction. Based on the Segment Anything Model (SAM), the image containing spill objects is processed to extract all potential target regions containing spill objects. The specific steps are as follows:
[0050] b1. Initialize the SAM model, determine parameters such as the number of sampling points and the minimum mask area based on the image pixels and sharpness, and automatically generate a mask for images containing projectiles;
[0051] b2. Calculate the IoU value. Based on the size of the projectile, the maximum mask area is individually divided. The mask area is calculated sequentially. Areas smaller than the maximum mask area are set as potential target areas. The bounding boxes and coordinates of these areas are recorded, and then the IoU value is calculated.
[0052] The formula for calculating IoU value is shown below:
[0053]
[0054] in:
[0055] The union is the area of the predicted bounding box compared to the actual bounding box.
[0056] intersection is the area of the part where the two frames intersect;
[0057] b3. Initially screen potential target regions. Use the non-maximum suppression algorithm to filter the potential target regions generated in the previous step and select regions with IoU values higher than the preset threshold.
[0058] Step 3: Highway spill classification. Using a contrastive language-image pre-trained model, visual embeddings of potential target regions are matched with text embeddings from an open vocabulary list. Confidence and similarity metrics are used to identify and classify spill targets. The specific steps are as follows:
[0059] c1. Projectile target recognition and localization: The image encoder of the Contrastive Language-Image Pre-training (CLIP) model is called to convert the latent target into a visual embedding, and the text encoder of the CLIP model is called to convert the open vocabulary list into a text embedding. The visual embedding of the latent target region and the text embedding of the open vocabulary list are matched through the contrastive language-image pre-training model.
[0060] c2. Combine the region bounding box and coordinates to calculate the confidence score and cosine similarity, and filter out the target region, i.e. the spilled object region, from the potential target region. At the same time, add the newly detected spilled object category to the open vocabulary list constructed in step one.
[0061] c3. Classification of spillage on highways: Match the target area obtained in step c2 to the open vocabulary list, calculate the probability of belonging to the category, and quantitatively analyze the semantic network between the most likely categories to determine the category of spillage corresponding to the target area.
[0062] Step 4: Highway debris warning. Record the type, material, size, and other properties of the debris, as well as their location and the time of fall. Return this information to the road monitoring terminal to assist in issuing different levels of alarm information.
[0063] In specific implementation, in this invention, the open vocabulary list constructed in step one comes from open source data on the Internet and new categories generated during the testing process. The initial category contains 1423 categories of spills, mainly including: (1) garbage: including cardboard, plastic, bottles, wooden boards, branches and leaves, food packaging, etc.; (2) goods: including shelves, express delivery, animals and other objects commonly transported on highways; (3) dust: including dust particles generated due to construction or vehicle crushing, etc.
[0064] In specific implementation, in this invention, when extracting potential target regions in step two, based on the image pixel size of 1920*1080 pixels, the maximum mask area is set to 1000 pixels, the number of sampling points on each side of the image is 64, the minimum mask region area is 100 pixels, and the IoU threshold for the non-maximum suppression process is 0.1. Other parameters remain at their default values, specifically including the prediction process IoU threshold of 0.86, the stability score threshold of 0.92, the number of cropping layers of 1, and the scaling factor of 2.
[0065] In specific implementation, in step three of this invention, the top ten most likely categories of spilled material are selected as references. By analyzing the semantic network behind these categories, the categories and properties of the spilled material are extracted, ensuring the accuracy and reliability of the auxiliary information.
[0066] In specific implementation, this invention does not require training or fine-tuning of the SAM and CLIP models for the projectile scenario. Instead, pre-trained models are used directly. The SAM model uses automatic search, and the CLIP model uses the ViT-B / 32 model, which has good generalization and transfer learning capabilities.
[0067] In specific implementation, the image data used in this invention includes daytime scenes and nighttime scenes. This invention focuses on small-scale spill scenarios and extreme situations such as nighttime glare and vehicle obstruction.
[0068] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms, and other features not explicitly described in this technical solution are considered common technical features disclosed in the prior art.
[0069] Example 1
[0070] This embodiment presents a method for detecting and classifying highway debris based on open vocabulary detection. (See also...) Figure 1 It includes the following steps:
[0071] Step 1: Based on common types of spillage, construct an open vocabulary list for highway spillage. The specific steps are as follows:
[0072] a1. First, extract keywords related to spilled materials from highway surveillance videos, related reports, and other data. The keywords mainly include the name, color, shape, material, and state of the spilled materials to ensure that all possible types of spilled materials are covered. The initial categories contain 1423 types of spilled materials, mainly including: (1) garbage: including cardboard, plastic, bottles, wooden boards, branches and leaves, food packaging, etc.; (2) goods: including shelves, express delivery, animals, and other objects commonly transported on highways; (3) dust: including dust particles generated by construction or vehicle crushing, etc.
[0073] a2. Based on the nature of the spilled material, classify the keywords and add synonyms and near-synonyms for each keyword;
[0074] a3. Construct a relationship network between each keyword and other words, including hierarchical relationships, relevance, antonyms, etc., and use the characteristics of the spilled object such as color, shape, material, and scale to establish a semantic relationship network between words.
[0075] Step Two: As Figure 2 As shown in (a), the automatic Segment Anything Model (SAM) is selected for potential projectile target region extraction. The image containing projectiles is processed to extract all possible target regions containing projectiles. The specific steps are as follows:
[0076] b1. Initialize the SAM model, such as Figure 3 The image data shown includes daytime and nighttime scenes. This invention focuses on small-scale debris scenes and extreme situations such as nighttime glare and vehicle obstruction. Based on image pixels and sharpness, the number of sampling points is determined to be 64, the minimum mask area is 100 pixels, and other parameters are as follows: prediction process IoU threshold is 0.86, stability score threshold is 0.92, number of cropping layers is 1, and scaling factor is 2. Masks are automatically generated for images containing debris.
[0077] b2. Calculate the IoU value. Based on the size of the projectile, determine the maximum mask area to be 1000 pixels. Calculate the mask area in sequence. Set the areas smaller than the maximum mask area as potential target areas. Record the bounding boxes and coordinates of these areas, and then calculate the IoU value.
[0078] The formula for calculating IoU value is shown below:
[0079]
[0080] in:
[0081] The union is the area of the predicted bounding box compared to the actual bounding box.
[0082] intersection is the area of the part where the two frames intersect;
[0083] b3. Initially screen potential target regions. Use a non-maximum suppression algorithm to filter the potential target regions generated in the previous step, selecting regions with bounding box IoU values higher than 0.1, such as... Figure 4 Each image shown can filter out approximately 250 potential target regions;
[0084] Step 3: Classify spilled materials from highways, such as... Figure 2 As shown in (b), the ViT-B / 32 model of the pre-trained Contrastive Language-Image Pre-training (CLIP) is used to match the visual embedding of the potential target region with the text embedding of the open vocabulary list. The identification and classification of the projectile target are achieved through confidence and similarity key indicators. The specific steps are as follows:
[0085] c1. Projectile target recognition and localization: The image encoder of the CLIP model is called to convert the latent target into a visual embedding, and the text encoder of the CLIP model is called to convert the open vocabulary list into a text embedding. Through a contrastive language-image pre-trained model, the visual embedding of the latent target region is matched with the text embedding of the open vocabulary list.
[0086] c2. Combine the region bounding box and coordinates to calculate the confidence score and cosine similarity, and filter out the target region, i.e. the spilled object region, from the potential target region. At the same time, add the newly detected spilled object category to the open vocabulary list constructed in step one.
[0087] c3. Classification of spilled materials on highways, such as Figure 5 As shown, the target region obtained in step c2 is matched to the list of open words. By calculating the probability of the category, and quantitatively analyzing the semantic network between the top 10 categories, the category of the spilled material corresponding to the target region is determined.
[0088] Step 4: Highway debris warning. Record the type, material, size, and other properties of the debris, as well as their location and the time of fall. Return this information to the road monitoring terminal to assist in issuing different levels of alarm information.
[0089] Application Example 1
[0090] An experiment was conducted to demonstrate the method of this invention, extracting 28,805 images (1920*1080 pixels each) from a highway surveillance video. Of these videos, 35 traffic accidents were related to debris spilled on the highway, with 4,007 frames containing at least one piece of debris.
[0091] 4007 images were input into the SAM model. An image encoder generated masks to segment objects, and potential target regions were extracted through comparison. On average, approximately 250 potential target regions were extracted, with a detection rate of 78% for projectiles. Then, the acquired potential target regions were processed image by image and output into the CLIP model. Combined with an open vocabulary list of projectiles, the top 10 projectile categories were obtained as a reference, achieving a classification accuracy of 93%.
[0092] The above description of the embodiments is provided to enable those skilled in the art to understand and use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the invention should be within the protection scope of the present invention.
Claims
1. A method for detecting and classifying highway spills based on open vocabulary detection, characterized in that, Includes the following steps: S1: Construct an open vocabulary list for highway spills based on common spill types; S2: The image is segmented. Based on the calculated mask size, the potential target regions of the projectiles are determined and obtained. During the extraction of the potential target regions, the image containing the projectiles is processed based on the SAM model to extract all possible target regions containing the projectiles. S3: By using a contrastive language-image pre-trained model, the visual embedding of the potential target region is matched with the text embedding of the open vocabulary list. By calculating the confidence index, the projectile target is accurately extracted from the potential target region. Then, by calculating the similarity index, each extracted projectile is classified. In S3, the specific process includes: c1. Projectile target recognition and localization: The image encoder of the CLIP model is called to convert the latent target into a visual embedding, and the text encoder of the CLIP model is called to convert the open vocabulary list into a text embedding. Through a contrastive language-image pre-trained model, the visual embedding of the latent target region is matched with the text embedding of the open vocabulary list. c2. Combine the region bounding box and coordinates to calculate the confidence score and cosine similarity, and filter out the target region, i.e. the spilled area, from the potential target region. At the same time, add the newly detected spilled category to the open vocabulary list constructed in S1. c3. Classification of spillage on highways: Match the target area obtained in step c2 to the open vocabulary list, calculate the probability of belonging to the category, and quantitatively analyze the semantic network between the most likely categories to determine the category of spillage corresponding to the target area. In c3, the top ten most likely categories of spilled material are selected as a reference. By analyzing the semantic network behind the selected categories, the categories and properties of spilled material are extracted, ensuring the accuracy and reliability of the auxiliary information.
2. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 1, characterized in that, In S1, the specific process includes: a1. Extract keywords related to spilled materials from highway surveillance videos and publicly reported data; a2. Based on the nature of the spilled material, classify the keywords and add synonyms and near-synonyms for each keyword; a3. Construct a network of relationships between each keyword and other words to build an open vocabulary list for highway spills.
3. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 2, characterized in that, In a1, the keywords include the name, color, shape, material, and state of the spilled material.
4. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 2, characterized in that, In a3, the relational network includes hierarchical relationships, relevance, and antonymous relationships, and uses the color, shape, material, and scale features of the spilled object to establish a semantic relational network between words; In A3, the list of open vocabulary is built from open-source data on the internet and new categories generated during testing. The initial categories include: garbage, goods, and dust.
5. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 1, characterized in that, In S2, the specific process includes: b1. Initialize the SAM model, determine the number of sampling points and the minimum mask area based on the image pixels and sharpness, and automatically generate a mask for images containing projectiles; b2. Calculate the IoU value. Based on the size of the projectile, the maximum mask area is individually divided. The mask area is calculated sequentially. Areas smaller than the maximum mask area are set as potential target areas. The bounding boxes and coordinates of these areas are recorded, and then the IoU value is calculated. b3. Initially screen potential target regions. Use the non-maximum suppression algorithm to filter the potential target regions generated in b2, and select regions with IoU values higher than the preset threshold.
6. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 5, characterized in that, In b2, the formula for calculating the IoU value is as follows: in: This represents the area of the predicted bounding box compared to the actual bounding box. This represents the area of the intersection of the two frames.
7. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 1, characterized in that, In S2 and S3, pre-trained models are used directly. The SAM model uses automatic search, and the CLIP model uses the ViT-B / 32 model. In S2 and S3, the image data used includes both daytime and nighttime scenes.
8. The method for detecting and classifying highway spills based on open vocabulary detection according to claim 1, characterized in that, It also includes S4: generating early warning information for road sections where debris is found, and feeding back the early warning information, the location and type of debris to the road monitoring terminal.
Citation Information
Patent Citations
Road thrown object detection method based on contrast clustering self-learning
CN114782891A
Road surface thrown object detection method based on joint multi-task model
CN116311161A
