A residual attribute prompt driven detection and incremental learning method for unknown thrown objects

Abstract

The application discloses a residual attribute prompt driven detection and incremental learning method for unknown scattering objects, aiming to solve the problems of missing detection, false detection and lack of self-learning ability of existing inspection on training set outside scattering objects. When the inspection vehicle or unmanned aerial vehicle enters the working area, start the visible light camera to collect real-time video stream; use the context residual learning module to detect the abnormal area of the video image, generate the residual energy graph to locate the potential unknown scattering object; based on the shape, texture and reflectivity statistics of the candidate frame area, construct the attribute vector and automatically generate the descriptive semantic prompt word, the prompt and the preset word library are fused through the gate weight to drive the open visual-linguistic segmentation, output the boundary information and semantic label of the scattering object; through the adaptive incremental learning module, the new detection sample is updated and knowledge playback with few samples, realizing the dynamic expansion and continuous evolution of the knowledge base. The application realizes the discovery of unknown targets through residual detection, completes semantic recognition and boundary extraction through open segmentation, and continuously expands the detection category through adaptive incremental learning, constructs a 'discovery-recognition-learning' closed loop mechanism, thereby significantly improving the processing ability of road inspection on unknown scattering objects in complex environment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent transportation technology, and in particular, it is a residual attribute prompt-driven detection and incremental learning method for unknown spilled materials. Background Technology

[0002] In scenarios such as road inspection, intelligent transportation, and assisted driving, spilled debris has a direct impact on driving safety. Existing deep learning-based object detection methods are typically trained based on the closed set assumption, and can only reliably identify categories that appear in the training set. When real-world scenarios involve unknown categories or appearance variations not covered in the training set, false negatives or missed detections are likely to occur. Furthermore, road scenarios exhibit strong spatiotemporal variability and long-tail distribution characteristics: different weather conditions, road types, and camera orientations can all lead to significant domain shifts, making the generalization ability of relying on a single detector insufficient.

[0003] On the other hand, existing methods mostly use bounding boxes as the main output, making it difficult to perform pixel-level segmentation and geometric constraints on projectiles, thus affecting subsequent risk assessment. While research on open set or anomaly detection for unknown targets can identify anomalous regions, it often lacks tight coupling with semantic understanding, pixel segmentation, and continuous learning, making it difficult to form an engineering closed loop. Introducing new categories into existing networks usually requires re-labeling and full training, which is costly, slow in iteration, and faces the problem of catastrophic forgetting. At the same time, practical deployments often rely on edge devices, which are limited by computing power and power consumption, placing higher demands on the real-time performance and resource consumption of the algorithms. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a residual attribute-driven detection and incremental learning method for unknown spills.

[0005] The technical solution to achieve the purpose of this invention is: a residual attribute hint-driven detection and incremental learning method for unknown spilled materials, comprising the following steps:

[0006] S1. Use a visible light camera to collect road information images;

[0007] S2. Input the road information image into the anomaly detection module. The anomaly detection module generates a residual energy map based on the context residual information and extracts the candidate box region of potential unknown spills according to a preset or adaptive threshold.

[0008] S3. Input the candidate box region into the residual attribute prompt generation module, extract at least one statistical feature from the shape, texture, and reflectivity of the candidate box region, calculate the confidence of the candidate box region, and generate descriptive semantic prompt words;

[0009] S4. Perform pixel-level segmentation and open-vocabulary semantic recognition on the local sub-image output by the anomaly detection module to determine the segmentation mask, semantic label and confidence level of the spilled object;

[0010] S5. Project the segmentation mask onto the bird's-eye view coordinate system, calculate the risk percentage in combination with the drivable area mask, generate the risk level and trigger a graded alarm.

[0011] S6. Perform data entry, quality assessment, and deduplication on novel spill samples of unknown categories to build a dynamic sample library;

[0012] S7. Based on a dynamic sample library, the joint model consisting of the anomaly detection module and the open vision-language segmentation module is updated with few samples through an adaptive incremental learning module; and the updated model parameters are synchronized to the edge device.

[0013] Compared with the prior art, the present invention has the following significant advantages:

[0014] (1) Candidate boxes are generated by context residuals, and then semantic judgment and pixel-level segmentation are achieved by visual-language alignment. Even if a new type of object is not included in the training set, it can be identified as an "object", effectively avoiding missed detection.

[0015] (2) Introducing the “residual attribute prompt generation” mechanism: Descriptive semantic prompts are automatically generated from the statistical features of the candidate box region and then fused with the preset word library according to the gate weight to drive segmentation, reducing the dependence on the manual word library and improving the accuracy and robustness of the identification of unknown spilled objects.

[0016] (3) New samples are automatically stored in the database, and the model parameters are updated through incremental training with a small number of samples, distillation and experience playback, and then transmitted back to the edge device to realize the online closed loop of "discovery-learning-update" and reduce the risk of full retraining and forgetting.

[0017] (4) In the output stage, the mask is mapped to the bird's-eye view coordinate system and geometrically integrated with the drivable area. The risk ratio is calculated and the path is determined according to the threshold. It supports hierarchical early warning for easy scheduling and handling.

[0018] (5) The calculation range is narrowed by using candidate boxes as a priori, and segmentation and image-text alignment are only performed in suspected areas. Combined with a lightweight decoder and cloud collaboration, it is suitable for deployment on edge devices.

[0019] (6) Through multi-scale residual measurement, cue fusion and mask-level suppression, it effectively resists sudden changes in illumination, rain and fog reflection and jitter interference; the cross-frame consistency mechanism ensures stable output and reduces false detection and flicker.

[0020] (7) Input and output are standardized arrays and mask indices, and thresholds and prompts can be adaptively adjusted; each module can be replaced, which is convenient for cross-platform deployment and long-term maintenance.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention.

[0023] Figure 2 This is a diagram of the architecture of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and specific examples:

[0025] The present invention is conceived as a residual attribute-driven detection and incremental learning method for unknown spilled materials, the specific steps of which are as follows:

[0026] S1. Use a visible light camera to collect road information data.

[0027] The process involves acquiring road video streams using visible light cameras mounted on inspection vehicles or drones, resulting in a time-ordered image sequence.

[0028] Start the visible light camera to acquire a video stream at 1080P / 30fps. Ensure it has high dynamic range (HDR) and good night vision capabilities to adapt to different lighting conditions. Decode the video frames and write them to a circular buffer, along with timestamps and coordinate information. Perform distortion correction, color normalization, and size unification on the input frames, and record the necessary homography matrix H within the camera for subsequent BEV fusion. If severe jitter exists, retain a 3-5 frame sliding window for subsequent temporal consistency evaluation.

[0029] S2. Input the acquired image sequence into the anomaly detection module, generate a residual energy map based on the contextual residual information, and extract potential unknown spill candidate box regions according to a preset or adaptive threshold. The anomaly detection module uses a lightweight backbone network. The specific steps for extracting potential unknown spill candidate box regions are as follows:

[0030] S2.1: First, the current input image is fed into the lightweight backbone network for forward inference. Since the backbone network itself contains hierarchical convolutions and downsampling structures, it naturally generates three sets of feature maps with resolutions approximately equal to the original image, 1 / 2, and 1 / 4 of the original image during a single inference process. Multi-scale features are... ;

[0031] To determine whether the current features are abnormal, it is also necessary to construct corresponding normal reference features. These reference features are not directly generated by a single network layer, but rather through an "offline anchor point + online adaptation" approach. The offline portion comes from a large number of images of normal roads without spilled materials collected before deployment. After extracting features from these images through the same backbone network, the median value is taken at the pixel level to obtain a relatively stable background template, ensuring a reliable initial reference even when historical frames are lacking.

[0032] The reference features are primarily updated online: an exponential moving average (EMA) is performed on features from the most recent approximately 3 seconds, allowing them to gradually converge adaptively with changes in lighting, weather, and road material, and the update speed is automatically adjusted according to the scene domain. To prevent anomalies from being written into the reference, an EMA update is skipped when the suspected anomaly area ratio in the current frame is >0.20%; if this occurs in multiple consecutive frames, it reverts to the sliding window median reference and temporarily increases the offline reference weight. The final output includes three-scale features and matching references, along with metadata such as domain labels and freeze / revert states, for direct use in the residual energy measurement of S2.2. Normal reference features. Aggregate exponential moving averages from contiguous samples without spillage and the background over a moving time window. Scale weights are... .

[0033] S2.2: After obtaining the current features and their corresponding reference features, it is necessary to assess the degree of difference between them to determine which areas may deviate from the normal road condition. To this end, feature residuals are calculated at three scales and considered as fundamental evidence of potential anomalies. Specifically, pixel-wise and channel-wise differences are performed on the features at each scale, and the squared norm is taken to amplify large deviations while naturally suppressing small perturbations. Calculate the residual energy map:

[0034]

[0035] In the formula, For multi-scale features, For reference features of normal roads, For scale weights, s Indicates the feature scale, y is the horizontal coordinate in the feature map, and y is the vertical coordinate in the feature map;

[0036] The multi-scale residual information is structurally complementary: high-resolution features are more sensitive to subtle texture abrupt changes, such as small glass fragments or small-volume spills; low-resolution features focus more on larger-scale structural anomalies such as shape and regional consistency. Therefore, weights (0.5, 0.3, 0.2) are assigned to the three scales, and they are synthesized into a weighted residual energy map. To avoid the influence of random noise on the results, lightweight smoothing and normalization are performed on the residuals at each scale before fusion to ensure that the value ranges of different scales remain consistent. The final output residual energy reflects the distribution of anomaly probability in the overall space for the current frame.

[0037] S2.3: Combine each pixel value in the residual energy map obtained in S2.2 with the binarization threshold. Comparison: If If a pixel is positive, it is marked as 1 (indicating a potential anomaly); otherwise, it is marked as 0. This yields the initial binary residual mask.

[0038] Binarization threshold Defined as

[0039]

[0040]

[0041] This yields a binary residual mask. Using a 3×3 square structuring element, opening and closing operations are performed sequentially, followed by connected component filtering to remove isolated small points or fine line noise. The number of small connected components... (default When the connection is open, it is iterated once more for both opening and closing to enhance noise suppression. Each connected region is transformed into a minimum bounding rectangle as a candidate box. At the same time, the shape of the connected component in the binary mask is preserved as a coarse mask. Set a lower limit for the area. (Based on 640×640 dimensions) and upper limit of aspect ratio Filter out isolated noise and stripe artifacts. Output the set of bounding boxes for candidate boxes. With coarse mask .

[0042] S2.4: To reduce the fragmented detection of the same object at different scales, this step performs cross-scale consistency fusion on candidate boxes from three scales. Specifically, candidate boxes from multiple scales at the same time are matched pairwise: if the IoU of any two bounding boxes at different scales is ≥0.30, they are considered the same entity, and their corresponding masks are fused using a union set and hole filling is performed (preferably using a distance transform threshold method, with a threshold preferably 0.7) to restore the coherent shape of the target. For objects located in boundary regions (e.g.) For suspected matches, this step introduces temporal consistency voting: referring to the corresponding region in the previous frame, if the IoU with the previous frame reaches a threshold (preferably ≥0.30), it is also considered mergingable. After fusion, if multiple fragments still exist in the mask, only the connected component with the largest area is retained to avoid false merging; at the same time, the bounding box of the candidate box is proportionally expanded (preferably 6%) and written back to facilitate the preservation of necessary context during the next pruning step. After this step, the candidate box has a more stable shape and more complete boundaries at different scales and in adjacent frames, significantly reducing the omissions and jitter in subsequent segmentation stages.

[0043] S2.5: After obtaining the candidate bounding boxes and their corresponding coarse masks, it is necessary to further determine the priority of these candidate box regions and crop them into independent sub-images for subsequent analysis. First, calculate the intensity score of the candidate box based on the residual energy within each coarse mask region. Specifically, calculate the average residual energy of the pixels covered by the mask to obtain...

[0044]

[0045] This value reflects the degree to which the candidate box region deviates from the normal background. Then, all candidate boxes are processed by... Sort them from highest to lowest, and select the top 20 as the areas that need to be focused on (i.e., ).

[0046] To ensure sufficient contextual information for subsequent attribute inference and incremental detection, the boundaries of the first 20 candidate boxes are expanded proportionally before cropping, with a default expansion ratio of 8%. If the shorter side of a candidate box is less than 24 pixels, it indicates that the region is too small and easily affected by neighboring regions; in this case, the expansion ratio is increased to 12% to ensure that more background cues are preserved in the cropped image. If the expanded region exceeds the original image boundary, mirror padding is used to fill it in and maintain content continuity. Finally, the first 20 candidate boxes are cropped to generate a local sub-image. , as the input of S4.

[0047] S3: Residual Attribute Hint Generation

[0048] A residual attribute hint generation module is introduced. This module takes a local sub-image as input and calculates at least one or a combination of statistical features such as shape, texture, and reflectivity (including but not limited to: area, perimeter, aspect ratio, rectangularity, roundness, convex hull ratio, skeleton length and number of branches, Sobel edge density, LBP histogram entropy, mean difference and variance ratio of mask inside and outside residuals, HSV highlight pixel ratio and saturation statistics). It also checks whether the region behaves consistently over time (3-5 frames before and after), thereby assigning a reliability score, i.e., conf, to the candidate box region.

[0049] The statistical features such as geometry, texture, and reflectivity extracted from local sub-images are normalized to ensure consistent numerical ranges, and then concatenated into vectors to input a lightweight multilayer perceptron (with ≤0.1M parameters) to obtain attribute scores for multiple labels such as texture, reflection, and color, and automatically generate descriptive semantic prompts of no more than six words.

[0050] S4. Perform pixel-level segmentation and open-vocabulary semantic recognition on the local sub-image output by the anomaly detection module to accurately define the boundary of the spilled material and assign it semantic labels.

[0051] S4.1: Construct an automatic suggestion set N (attribute phrases) from the descriptive semantic prompts generated by the residual attribute prompt generation module; build a lightweight pre-defined vocabulary V={debris,road_object,tire,box} as backup anchors. Here, debris represents road debris or irregular scattered objects, road_object represents general foreign objects on the road, tire represents tire-like obstacles, and box represents boxes, cartons, or regular block-shaped objects.

[0052] Based on the local subgraph confidence score (conf) output by S3, the gating weights are dynamically calculated. This is used to balance the contributions of the two sources of prompts:

[0053]

[0054] in, For the Sigmoid function, This is a scaling hyperparameter; the default value is 3.0. In low-visibility scenarios such as nighttime or rain / fog, it will... The upper limit has been lowered by 0.1 to enhance reliance on a reliable preset thesaurus.

[0055] All prompts from the auto-suggestion set N and the preset vocabulary V are combined to form a unified candidate box pool. For each prompt t in the candidate box pool, its fusion score is calculated:

[0056]

[0057] in, It is a local subgraph Cosine similarity to the cue word t It is an image The average similarity with the entire auto-suggestion set N. The top-K suggestions (default K=3) with the highest scores are selected to form a fused suggestion set used to drive segmentation. .

[0058] S4.2: Filter the similarity threshold of valid prompt words and calculate the fused prompt set. Semantic association strength with local subgraphs:

[0059] Segmentation trigger threshold These are predefined constants, determined through offline grid search on the validation set with the objective of maximizing the F1 score.

[0060]

[0061] Where P is precision and R is recall.

[0062] The default value is: daytime scene. Nighttime scenes Rain and fog scenes When scene domain recognition is not performed, a unified approach is adopted. .

[0063] Local subgraphs and fusion hint set Each cue word t in the text is input into the visual encoder and the text encoder to obtain its feature vector. and And calculate the cosine similarity between the two as their semantic association score:

[0064]

[0065] This set of similarity scores will be used to drive segmentation in S4.3 and determine the final semantic labels in S4.4. .

[0066] S4.3: Input the fused cue set and the local subgraph into the open visual-language segmentation model to obtain...

[0067] The open visual-language segmentation module in this embodiment is centered on a segmentation decoder that accepts multimodal cues. This segmentation decoder can be based on existing visual-language segmentation models (e.g., Transformer-based mask decoders, lightweight decoders based on the SAM model). These existing models have already implemented the following: input image and text (or point, box) cues, output pixel-level masks of the corresponding targets.

[0068] In this invention, the specific workflow of the segmentation decoder is as follows:

[0069] The fusion hint set obtained in S4.1 The local subgraph obtained with S2.5 For the shared input open-ended vision-language segmentation model, this step introduces one or more of the following visual cues to improve segmentation accuracy and stability:

[0070] Tip: In coarse mask The energy peak position was sampled at 3 internal points;

[0071] Box hint: Use the expanded candidate box ;

[0072] Coarse mask hint: Use directly Based on the above hints, the segmentation decoder performs segmentation on the subgraph. Output binary mask .

[0073] S4.4: Mask set generated for each prompt word Mask-level non-maximum suppression (NMS) is performed to remove a large number of overlapping and duplicate segmentation results. NMS uses the intersection-union ratio (IUU) as an overlap metric. When the IUU between two candidate masks is greater than a preset threshold, the overlap is reduced. At this point, only the mask regions with higher scores are retained. For the multiple retained valid masks, their weighted union is further calculated to generate the final segmentation result. ;

[0074] Subsequently, the semantic scores corresponding to different prompt words were compared. (This indicates the semantic matching strength between the prompt word and the local subgraph) and selects the prompt word with the highest score as the category label for the current candidate region. .

[0075] When all prompt word scores are below the segmentation trigger threshold But residual strength fraction (default When the candidate box intensity is in the top 30% quantile, the output label "unknown-debris" indicates that the region is marked as an unknown debris, and the corresponding segmentation mask is preserved. .

[0076] S4.5: Calculate the overall confidence level of the test results For subsequent quality control and alarm classification:

[0077]

[0078] in It is Sigmoid. , To normalize the score for the contrast between the inside and outside of the mask, for Linear normalization to The final output is a complete record of the spilled material:

[0079]

[0080] in Indicates the first The bounding box corresponding to each local subgraph. Indicates the first The segmentation mask obtained after segmenting a local subimage. Indicates the first Category labels corresponding to each local sub-image Indicates the first Target confidence of a local subgraph Indicates the first The residual intensity score of each local subplot.

[0081] S5. Apply the segmentation mask obtained in the previous steps. The risk percentage is quantified by projecting the homography matrix onto the bird's-eye view coordinate system. Then, by calculating the geometric intersection of the homography matrix with the drivable area mask, a stable and reliable risk level and early warning information are output, thus completing the decision-making closed loop from "perceiving objects" to "assessing threats".

[0082] S5.1: Using the calibration matrix H to... Projected onto BEV, The unified BEV grid resolution is... , covering forward If the calibration self-test error is... This triggers online H re-evaluation but does not block inference.

[0083] S5.2: Use a semantic segmentation model (such as DeepLab, PSPNet, etc.) to segment the camera input image, classifying pixels into road, non-road, etc. Set road pixels to 1 and non-road pixels to 0 to obtain the drivable area mask L. Define the risk percentage.

[0084]

[0085] Set threshold The calibration is performed using lane width and vehicle envelope statistics, by default. .when When the system detects an abnormal target, it assumes that the vehicle has occupied the drivable area and marks the target as debris scattered along the path.

[0086] To reduce false triggering due to jitter, uplink and downlink threshold hysteresis is used: An entry threshold is set. Only when the risk proportion is greater than Only when the system encounters spilled material within the path will it enter the state of exiting the threshold. Only when the risk percentage is less than Only when the system detects spilled material within the path will it exit the state; and it will adaptively amplify the data based on the vehicle speed v. .

[0087] S5.3: Set tiering rules: High risk Or may overlap with the center of the lane; medium risk Low risk Cross-frame temporal consistency threshold Only if three consecutive frames satisfy consistency and the intersection-union ratio of two masks in adjacent frames is... The final alarm and report are triggered in a timely manner. Multi-target matching employs the Hungarian algorithm, which has a high cost. .like If confidence decay is triggered in nighttime, rainy, or foggy environments, the required number of consecutive frames is increased from 3 to 5. The alarm log contains <frame_id,t,GPS,lane ID,speed v,R_i^lane,level>.

[0088] S6. The sample library receives the detection results from the open vision-language segmentation unit and performs a storage and quality control process on samples belonging to the novel type of spill, specifically including:

[0089] S6.1: Set up an ingestion mechanism to automatically collect anomalous target samples with low confidence or unknown categories. When the category label of the candidate region... Or target confidence level (Default confidence threshold) When a data entry is triggered, the data is stored in the database, and sample information is recorded during the data entry process.

[0090] in This indicates the video frame number where the sample is located, t represents the timestamp corresponding to the sample, and GPS represents the geographical location information corresponding to the abnormal sample. Indicates the first The bounding boxes corresponding to each candidate region Indicates the first The final target mask obtained after segmenting the candidate regions. Indicates the first The category labels corresponding to each candidate region Indicates the first The target confidence level of each candidate region Indicates the first The residual intensity score of each candidate region.

[0091] S6.2: Score the incoming samples: Take the sample quality score.

[0092]

[0093] in , , , where e is the entropy of the gray-level histogram of the mask region. .when (default (Remove).

[0094] S6.3: Clean up the incoming samples to avoid redundancy, maintain class balance, and prepare for incremental learning. The specific implementation is as follows:

[0095] Extracting global features Similarity

[0096]

[0097] Set similarity threshold .

[0098] For all satisfying Sample pairs that are highly similar are considered redundant samples.

[0099] Deduplication strategy: In this set of redundant samples, samples with a larger timestamp span (i.e., collected at different times) or more obvious appearance differences (such as higher image entropy or more unique color distribution) are retained first, and the remaining samples are removed.

[0100] S6.4: Auditing and Versioning

[0101] Samples entering the database are manually inspected and marked as "passed / reverted". The sample database is managed by date and version for easy traceability and rollback.

[0102] S7. Perform few-sample incremental updates on the joint detection model composed of the anomaly detection module and the open vision-language segmentation module to achieve continuous evolution of the knowledge base, specifically including:

[0103] S7.1: Samples a small batch of samples from the sample library with a ratio of "new sample: replay sample = 0.5:0.5". The replay sample covers the historical main class and multi-domain labels. The default batch size is 32, and the maximum iteration is 1000 steps or early stop.

[0104] S7.2: Parameter Settings:

[0105] In the incremental learning process, to balance the ability to learn new categories with the ability to retain historical knowledge, this embodiment uses a joint loss function to optimize the model during training. The joint loss function is defined as follows:

[0106]

[0107] in For the supervision loss of new class samples, cross-entropy / focus loss is used. For knowledge distillation loss, To replay the monitored items, For parameter regularization, the default weight is... Distillation temperature 2.0.

[0108] .

[0109] in For the first The importance weight of each parameter For the current model, the One parameter, For the old model One parameter.

[0110] S7.3: A layered freezing strategy is adopted for the trunk and neck, and a low learning rate is set (for...). Prioritize updating the category / mask and set a high learning rate for it. The optimizer uses AdamW, with weight decay set to... The mean intersection-union ratio and unknown recall are monitored simultaneously on the validation set. If there is no improvement after 10 consecutive validations, the training is terminated early.

[0111] S7.4: After training is completed, the new weights are sent back to the edge side through the parameter update channel; the edge devices first replace them with 10% traffic in a grayscale manner, and then switch to the whole system after passing the stability threshold (unknown recall ↑ and false positives ≤ old version false positives); if communication is abnormal, the previous version is maintained and the status to be updated is recorded.

[0112] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0113] It should be understood that, in order to simplify the present invention and help those skilled in the art understand its various aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or with reference to a single figure. However, the present invention should not be construed as including all features in the exemplary embodiments as essential technical features of the claims of this patent.

[0114] It should be understood that the modules, units, components, etc., included in the device of one embodiment of the present invention can be adaptively changed to be placed in a device different from that embodiment. Different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. A residual attribute hint-driven detection and incremental learning method for unknown spilled materials, characterized in that, Includes the following steps: S1. Use a visible light camera to collect road information images; S2. Input the road information image into the anomaly detection module. The anomaly detection module generates a residual energy map based on the context residual information and extracts the candidate box region of potential unknown spills according to a preset or adaptive threshold. S3. Input the candidate box region into the residual attribute prompt generation module, extract at least one statistical feature from the shape, texture, and reflectivity of the candidate box region, calculate the confidence of the candidate box region, and generate descriptive semantic prompt words; S4. Perform pixel-level segmentation and open-vocabulary semantic recognition on the local sub-image output by the anomaly detection module to determine the segmentation mask, semantic label and confidence level of the spilled object; S5. Project the segmentation mask onto the bird's-eye view coordinate system, calculate the risk percentage in combination with the drivable area mask, generate the risk level and trigger a graded alarm. S6. Perform data entry, quality assessment, and deduplication on novel spill samples of unknown categories to build a dynamic sample library; S7. Based on the dynamic sample library, the joint model composed of the anomaly detection module and the open vision-language segmentation module is updated with few samples through the adaptive incremental learning module; and the updated model parameters are synchronized to the edge device.

2. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 1, characterized in that, The anomaly detection module generates a residual energy map based on contextual residual information and extracts candidate regions for potential unknown spills according to preset or adaptive thresholds. The specific method is as follows: S2.1 A lightweight backbone network is used to extract multi-scale features of the image and construct reference features for normal roads. The reference features are obtained by fusing the median of offline no-spillage sample features with the exponential moving average of online sliding window features. S2.2 Calculate the residual energy map for each scale feature. The specific formula is as follows: In the formula, For multi-scale features, For reference features of normal roads, For scale weights, s Indicates the feature scale, y is the horizontal coordinate in the feature map, and y is the vertical coordinate in the feature map; S2.

3. Compare each pixel value in the residual energy map with the binarization threshold to determine the binary residual mask. Use square structuring elements to perform opening and closing operations on the binary residual mask in sequence and perform connected component filtering. Convert each connected region into a minimum bounding rectangle as a candidate box, while retaining the shape of the connected component in the binary mask as a coarse mask. S2.4: Fuse candidate boxes with multi-scale features; S2.5: Calculate the intensity score of the candidate box based on the residual energy in each coarse mask region, and crop the candidate boxes with the highest intensity scores to a set number of local sub-images.

3. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 2, characterized in that, The specific formula for calculating the intensity score of the candidate box based on the residual energy within each coarse mask region is as follows: In the formula, This is a residual energy map. To iterate and sum over all pixels in the coarse mask region, This represents the coarse mask region corresponding to the i-th candidate box. The number of pixels contained in the coarse mask region.

4. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 1, characterized in that, The specific method for performing pixel-level segmentation and open-vocabulary semantic recognition on the local sub-image output by the anomaly detection module to determine the segmentation mask, semantic label, and confidence level of the spilled material is as follows: S4.1: Construct an automatic suggestion set from descriptive semantic prompts; dynamically calculate gating weights based on the confidence of candidate box regions, and construct a unified candidate box pool from the automatic suggestion set and all prompts in the preset word library; The fusion score of each prompt word in the candidate box pool is calculated based on the gating weight, and the K prompt words with the highest scores are selected to form a fusion prompt set; S4.2: Calculate the semantic association strength between the fusion suggestion set and the candidate boxes; S4.3: Input the fused cue set and the local subgraph into the open vision-language segmentation model to obtain a binary mask; S4.4: Perform nonmaximum suppression on the binary mask to determine the category label and the corresponding segmentation mask; S4.5: Calculate the overall confidence score and output the segmentation mask, semantic label, and confidence score of the local subgraph.

5. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 4, characterized in that, The specific method for calculating the semantic association strength between the fusion suggestion set and the candidate boxes is as follows: Local subgraphs and fusion hint set Each cue word t in the text is input into the visual encoder and the text encoder to obtain the corresponding feature vector. and And calculate the eigenvectors and Cosine similarity as semantic association strength : 。 6. The residual attribute hint-driven detection and incremental learning method for unknown spills according to claim 4, characterized in that, The specific formula for calculating the overall confidence level is as follows: in It is Sigmoid. , To normalize the score for the contrast between the inside and outside of the mask, Residual strength fraction Linear normalization to The value of .

7. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 4, characterized in that, The specific formula for calculating the fusion score of each prompt word in the candidate box pool based on the gating weight is as follows: in, It is a local subgraph Cosine similarity to the cue word t It is an image The average similarity with the entire auto-suggestion set N, This is the gating weight.

8. The residual attribute hint-driven detection and incremental learning method for unknown spilled materials according to claim 1, characterized in that, The specific method for projecting the segmentation mask onto the bird's-eye view coordinate system, calculating the risk percentage in conjunction with the drivable area mask, generating a risk level, and triggering a graded alarm is as follows: S5.1: Use the calibration matrix to project the segmentation mask... Projecting onto the bird's-eye view coordinate system, we obtain S5.2: Use a semantic segmentation model to segment the camera input image, classifying pixels into roads and non-roads; set road pixels to 1 and non-road pixels to 0 to obtain the drivable area mask L; Calculate the risk percentage: When the risk percentage When the system determines that the corresponding abnormal target has occupied the vehicle's drivable area, it marks the target as debris scattered along the path. Uplink and downlink threshold hysteresis: Set entry threshold When the risk ratio is greater than When there is spilled material within the path, the exit threshold is reached. When the risk percentage is less than At that time, there is a state of spilled material within the exit path; and adaptive amplification is performed based on the vehicle speed v: . S5.3: Set classification rules based on risk percentage and trigger alarms when the set alarm conditions are met.

9. The residual attribute hint-driven detection and incremental learning method for unknown spills according to claim 1, characterized in that, The loss function for training the joint model consisting of the anomaly detection module and the open vision-language segmentation module is: in, The supervised loss for new category samples, Used to maintain the old category distribution For historical sample supervision terms in the playback buffer, For parameter regularization; hierarchical parameter freezing and elastic weight consolidation are used in the backbone network to suppress catastrophic forgetting: in For the first The importance weight of each parameter For the current model, the One parameter, For the old model One parameter.

Images