An online error sample exclusion method based on self-supervised visual representation
By comparing the similarity between self-supervised visual representations and error sample libraries, the false alarm problem of visual algorithms in new scenarios is solved, achieving stable visual algorithm output and low-cost false alarm suppression.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NEWLAND DIGITAL TECH CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-29
AI Technical Summary
Existing visual algorithms are prone to false alarms when deployed to new scenarios, and existing solutions are costly, complex to maintain, and difficult to adapt to rapidly changing real-world scenarios.
A self-supervised visual representation method is adopted to obtain the feature vectors of candidate targets online and compare them with the error sample library. False alarms are filtered out by the error sample library. The method combines approximate nearest neighbor retrieval and vector database to achieve rapid identification and suppression of false alarms.
Without changing the main model structure, it significantly reduces the false positive rate, improves the recall rate, reduces maintenance costs, and achieves stable visual algorithm output.
Smart Images
Figure CN122116094A_ABST
Abstract
Description
Technical Field
[0001] This invention is applied to the fields of computer vision and artificial intelligence, specifically an online error sample elimination method based on self-supervised visual representation. Background Technology
[0002] Visual algorithms often achieve ideal performance on data distributed in the same way as the training set. However, when deployed to new scenes not covered in the training set, out-of-distribution (OOD) data significantly amplifies the probability of false positives. Taking pedestrian detection as an example, when there are deviations in camera position, viewpoint, resolution, lighting, and weather conditions, the discrimination cues learned by the model during training will differ greatly from the real environment. Some non-target entities (such as tree trunks, animals, and bright light spots) may couple with the concept of "human" in local texture or shape, thus triggering stable and reproducible false positives. Similar problems are not limited to pedestrian detection; they are common in tasks such as vehicle detection and smoke detection: when the model encounters an unseen scene, it is more likely to treat "similar but not identical" objects as positive classes. The essence of this phenomenon is the generalization problem of visual algorithms.
[0003] Current engineering practices typically delegate the responsibility of reducing false positives to the model itself. This is achieved by expanding training data, designing more robust data augmentation, introducing hard sample mining, adjusting loss functions, or adopting more complex network structures to pursue gains on test set metrics. However, the pace and diversity of real-world scenarios far exceed the rhythm of training and deployment. New scenarios constantly generate new false positive categories, and offline retraining requires re-collecting, cleaning, and labeling data, followed by retraining, deployment, and validation. This iteration cycle often doesn't match the frequency of problem occurrence. To achieve timely suppression, some systems employ post-processing techniques such as threshold adjustment, region masking, and temporal rules. However, these rules are highly sensitive to scene and environmental details, requiring readjustment when migrating to new scenarios or time periods. Maintenance costs increase almost linearly with the number of scenarios, and the system is prone to fluctuating between false positives and false negatives, causing overall performance instability.
[0004] If we consider the false positive samples exposed in online algorithms as reusable "negative evidence," the problem takes a different approach. Many false positives are not random noise, but rather inductively categorized groups that share stable features in semantics, texture, edges, shape, or material. Instead of accumulating this negative evidence and adding it to the next round of training, we can transform it into searchable objects in the form of general visual features without changing the main model. Then, in subsequent inference stages, we can perform a rapid similarity check on each candidate alarm result: if the new candidate content is highly similar to historical false positives, it is directly identified as a false positive and suppressed in the current frame. This online filtering mechanism can transform "previously occurring false positives" into the ability to "immediately identify and block them in the next iteration," bypassing the lengthy retraining and release cycle and forming a solution that matches the speed of change in reality.
[0005] In recent years, self-supervised visual representations (such as training paradigms centered on contrast or distillation) have learned transferable global and local features without relying on task labels. These features are particularly robust in capturing "appearance similarity." Meanwhile, the development of near-nearest neighbor retrieval and vector databases has made millisecond-level retrieval on millions of vectors a common engineering practice, providing a feasible system foundation for "performing a false positive similarity check on each candidate." Therefore, an online false sample elimination framework decoupled from the main model can be constructed: during the operation of the visual algorithm, when an alarm result is confirmed as a false alarm by manual verification, its general visual features are immediately extracted and stored in a false sample library composed of vector spaces. In each subsequent inference, features are extracted synchronously for newly generated candidates and compared with vectors in the library for similarity; if the similarity exceeds a threshold, it is directly filtered. This mechanism does not conflict with offline retraining, but it establishes a hierarchical division of labor between the two: the online layer is responsible for "instant loss mitigation," while the offline layer is responsible for "optimization after accumulating false alarms over a period of time," significantly reducing recurring false alarms while ensuring stable recall.
[0006] The goal of this online error sample exclusion mechanism is not to replace learning with rules, but to fill the generalization gap through "experience accumulation + rapid retrieval". When faced with unseen scenarios, the main model still provides first-order discrimination, while the error sample library provides second-order verification. The two work together to transform the continuous emergence of false alarms in the real world into manageable and evolvable knowledge assets. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide an online error sample elimination method based on self-supervised visual representation, which addresses the shortcomings of the prior art.
[0008] To address the aforementioned technical problems, this invention provides an online error sample elimination method based on self-supervised visual representation, applied to the online inference process of visual algorithms, specifically including the following steps:
[0009] The system acquires the image or video frame to be processed, and infers from it using the main visual model, outputting at least one candidate target result, wherein the candidate target result includes at least the candidate region and its category and / or confidence level;
[0010] The candidate regions are cropped or extracted and input into a self-supervised visual feature encoding model to obtain candidate feature vectors corresponding to the candidate regions.
[0011] An approximate nearest neighbor search is performed in the error sample database, and the candidate feature vector is compared with the feature vector of the error samples in the database to obtain the similarity result.
[0012] The similarity result and threshold strategy are used to determine whether the candidate target result is a false alarm; when the similarity meets the preset condition, the candidate target result is marked as a false alarm and removed from the output result; otherwise, it is retained and output to subsequent processing.
[0013] When a candidate target result is confirmed as a false alarm by manual review, the false alarm feature vector of the corresponding region is extracted, and the false alarm feature vector and its metadata are written into the error sample library to achieve online incremental updates.
[0014] As one possible implementation, the main visual model is any one of an object detection model, a classification model, or a segmentation model, and the online error sample elimination method is executed without changing the structure and training parameters of the main visual model.
[0015] As one possible implementation, the feature vector output by the self-supervised visual feature encoding model is a global embedding vector, and the candidate feature vector and the false positive feature vector are normalized before being stored in the database and / or retrieved.
[0016] As one possible implementation, the similarity comparison further employs a cosine similarity measure, or the normalized feature vectors are subjected to an inner product measure to achieve retrieval equivalent to cosine similarity.
[0017] As one possible implementation, the error sample library further performs approximate nearest neighbor retrieval based on a vector index structure, which includes an inverted file and product quantization index structure and / or a hierarchical small-world graph index structure.
[0018] As one possible implementation, the codebook of the vector index structure and the coarse quantizer are further trained and solidified offline using representative samples.
[0019] As one possible implementation, the threshold strategy further includes at least one of a global threshold strategy and a subdomain adaptive threshold strategy; the subdomain adaptive threshold strategy sets different thresholds according to category, camera position, and / or time period.
[0020] As one possible implementation, the threshold strategy further involves periodically re-evaluating the threshold based on the runtime similarity distribution and cost function.
[0021] As one possible implementation, the metadata further includes at least one of the following: timestamp, scene identifier, inference threshold, category information, and source camera information.
[0022] An online error sample exclusion system based on self-supervised visual representation includes:
[0023] The inference module is used to call the main visual model to perform inference on the input image or video frame and output the candidate target result;
[0024] The feature extraction module is used to crop or extract the candidate regions corresponding to the candidate target results, and generate candidate feature vectors through a self-supervised visual feature encoding model.
[0025] The error sample library module is used to store the feature vectors of error samples and their metadata, and supports online incremental writing;
[0026] The similarity verification module is used to perform approximate nearest neighbor retrieval in the error sample library module to obtain similarity results, and to determine whether to mark the candidate target results as false alarms and filter them according to the threshold strategy;
[0027] The false alarm entry module is used to generate a false alarm feature vector and write it along with metadata into the error sample database module after the candidate target result is confirmed as a false alarm by manual review.
[0028] The present invention adopts the above technical solution and has the following beneficial effects:
[0029] By integrating this invention into existing vision algorithms, without adjusting the main model structure and training paradigm, it can quickly transform false alarms into reusable online filtering capabilities. Based on DINOv2's vits14, 384-dimensional normalized features are extracted. Combined with approximate nearest neighbor retrieval from a vector database, in representative pedestrian and non-motorized vehicle mixed traffic scenarios, recurring false alarms caused by factors such as reflective cones, billboards, and tree trunks are stably identified and promptly suppressed. The average false alarm rate is significantly reduced compared to the baseline, while the recall of the main model remains essentially unchanged, and the F1 score is considerably improved under the same threshold system. The overall end-to-end latency is in the millisecond range, allowing for long-term operation across multiple cameras and scenarios. Compared to solutions relying on frequent retraining, this invention replaces the high-cost training and deployment process with vector retrieval, enabling the system to quickly acquire "retraining-free" filtering capabilities after the first false alarm is detected. These capabilities automatically accumulate and improve over time, bringing substantial benefits simultaneously in terms of engineering maintenance, resource consumption, and performance stability. Attached Figure Description
[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0031] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0033] This invention provides an online error sample elimination method based on self-supervised visual representation, which is applied to the online inference process of visual algorithms, and specifically includes the following steps:
[0034] The system acquires the image or video frame to be processed, and infers from it using the main visual model, outputting at least one candidate target result, wherein the candidate target result includes at least the candidate region and its category and / or confidence level;
[0035] The candidate regions are cropped or extracted and input into a self-supervised visual feature encoding model to obtain candidate feature vectors corresponding to the candidate regions.
[0036] An approximate nearest neighbor search is performed in the error sample database, and the candidate feature vector is compared with the feature vector of the error samples in the database to obtain the similarity result.
[0037] The similarity result and threshold strategy are used to determine whether the candidate target result is a false alarm; when the similarity meets the preset condition, the candidate target result is marked as a false alarm and removed from the output result; otherwise, it is retained and output to subsequent processing.
[0038] When a candidate target result is confirmed as a false alarm by manual review, the false alarm feature vector of the corresponding region is extracted, and the false alarm feature vector and its metadata are written into the error sample library to achieve online incremental updates.
[0039] The main visual model can be any one of an object detection model, a classification model, or a segmentation model, and the online error sample elimination method is executed without changing the structure and training parameters of the main visual model.
[0040] The feature vector output by the self-supervised visual feature encoding model is a global embedding vector, and the candidate feature vector and the false positive feature vector are normalized before being entered into the database and / or retrieved.
[0041] The similarity comparison uses cosine similarity as a metric, or the inner product metric is used on the normalized feature vectors to achieve retrieval equivalent to cosine similarity.
[0042] The error sample library performs approximate nearest neighbor retrieval based on a vector index structure, which includes an inverted file and product quantization index structure and / or a hierarchical small-world graph index structure. The codebook and coarse quantizer of the vector index structure are trained offline and solidified using representative samples. The threshold strategy includes at least one of a global threshold strategy and a subdomain adaptive threshold strategy; the subdomain adaptive threshold strategy sets different thresholds according to category, camera position, and / or time period. The threshold strategy periodically re-evaluates the threshold based on the runtime similarity distribution and cost function.
[0043] The metadata includes at least one of the following: timestamp, scene identifier, inference threshold, category information, and source camera information.
[0044] An online error sample exclusion system based on self-supervised visual representation includes:
[0045] The inference module is used to call the main visual model to perform inference on the input image or video frame and output the candidate target result;
[0046] The feature extraction module is used to crop or extract the candidate regions corresponding to the candidate target results, and generate candidate feature vectors through a self-supervised visual feature encoding model.
[0047] The error sample library module is used to store the feature vectors of error samples and their metadata, and supports online incremental writing;
[0048] The similarity verification module is used to perform approximate nearest neighbor retrieval in the error sample library module to obtain similarity results, and to determine whether to mark the candidate target results as false alarms and filter them according to the threshold strategy;
[0049] The false alarm entry module is used to generate a false alarm feature vector and write it along with metadata into the error sample database module after the candidate target result is confirmed as a false alarm by manual review.
[0050] Example:
[0051] like Figure 1 As shown, this invention utilizes the feature extraction capabilities of a self-supervised visual model to extract high-dimensional representations from inductively categorized false positive samples and establishes an error sample library in a vector database. During the online inference process of the visual algorithm, features are simultaneously extracted for each candidate result and compared with the error sample library for similarity. When the similarity reaches a preset or adaptive threshold, it is directly identified as a false positive and filtered out, thereby significantly reducing false positives and stabilizing business performance without modifying the main model. For ease of verification and implementation, this invention uses the self-supervised model DINOv2 as the feature encoder, with the test model version ViT-S / 14. It is globally embedded as a general representation of candidate and false positive samples, and normalized cosine similarity is used for fast comparison in the vector space. This invention does not limit the specific implementation of the feature model; any model with stable and general visual representation capabilities can be used instead.
[0052] The system consists of three collaborative parts. The first part is the existing visual algorithm inference chain, which maintains the existing object detection, classification, or segmentation models and outputs candidate targets, categories, and confidence scores for input images or video frames. The second part is the error sample establishment chain. When a candidate result is confirmed as a false alarm by manual review, the system automatically crops the corresponding image region and calls DINOv2 forward propagation to obtain a fixed-length feature vector. Simultaneously, it records metadata such as timestamps, scene identifiers, inference thresholds, categories, and source camera positions, and then writes this information into the vector database for online incremental updates. The third part is the similarity verification chain. After each inference output candidate, the feature vector of that candidate is immediately extracted synchronously, and an approximate nearest neighbor query is initiated against the error sample database. The returned nearest neighbor similarity statistics are combined with a threshold strategy for judgment. When the similarity exceeds the threshold, the candidate is marked as a false alarm and directly suppressed in the current frame, removed from the final result stream; otherwise, it proceeds normally to post-processing and the business side.
[0053] To ensure real-time performance and high-concurrency retrieval, this invention employs the open-source vector retrieval library FAISS as the core of the vector database. Within the same framework, it configures the IndexIVFPQ (combined with inverted files and product quantization) and the IndexHNSW* series of indexes using hierarchical small-world graphs. The codebook and coarse quantizer are trained offline with representative samples using k-means and then solidified before deployment. Taking the 384-dimensional embedding output from DINOv2 (vits14) as an example, the system performs L2 normalization on the vectors during database entry and querying, and uses inner product metric to achieve retrieval equivalent to cosine similarity. The IVF path uses nlist / nprobe, and the HNSW path uses online adaptive adjustment via M / efSearch to dynamically balance recall and latency. The threshold can be set as a global constant or adapted for subdomains based on category, camera position, and time period, and periodically re-evaluated based on the runtime similarity distribution and cost function.
[0054] In terms of method selection, this invention uses DINOv2's ViT-S / 14 as the test model during the verification phase. Practice shows that this model's ability to capture appearance and semantic similarity in different scenarios, as well as its running speed, meet online requirements. Combined with IVF-PQ or HNSW, it can maintain millisecond-level query latency even with millions of samples. For applications requiring higher discriminative power, the system can also concatenate a small number of weak contextual features into the candidate features while maintaining the main scheme, thereby improving stability in complex contexts.
[0055] In summary, this invention compensates for the traditional false alarm solutions that rely on model retraining by using an online mechanism of "establishing an error sample library - similarity verification - online filtering". It transforms the exposed false alarms into real-time protection capabilities in subsequent operation, enabling visual algorithms to obtain more stable output quality with lower maintenance costs in multi-scenario and long-term applications.
[0056] The above description is an embodiment of the present invention. For those skilled in the art, any equivalent changes, modifications, substitutions and variations made in accordance with the teachings of the present invention without departing from the principles and spirit of the present invention should be covered by the present invention.
Claims
1. An online error sample elimination method based on self-supervised visual representation, applied to the online inference process of visual algorithms, characterized in that, Specifically, the steps include the following: The system acquires the image or video frame to be processed, and infers from it using the main visual model, outputting at least one candidate target result, wherein the candidate target result includes at least the candidate region and its category and / or confidence level; The candidate regions are cropped or extracted and input into a self-supervised visual feature encoding model to obtain candidate feature vectors corresponding to the candidate regions. An approximate nearest neighbor search is performed in the error sample database, and the candidate feature vector is compared with the feature vector of the error samples in the database to obtain the similarity result. The similarity results and threshold strategy are used to determine whether the candidate target result is a false alarm; When the similarity meets the preset conditions, the candidate target result is marked as a false alarm and removed from the output result; otherwise, it is retained and output to subsequent processing. When a candidate target result is confirmed as a false alarm by manual review, the false alarm feature vector of the corresponding region is extracted, and the false alarm feature vector and its metadata are written into the error sample library to achieve online incremental updates.
2. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The main visual model is any one of an object detection model, a classification model, or a segmentation model, and the online error sample elimination method is executed without changing the structure and training parameters of the main visual model.
3. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The feature vector output by the self-supervised visual feature encoding model is a global embedding vector, and the candidate feature vector and the false positive feature vector are normalized before being added to the database and / or retrieved.
4. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The similarity comparison uses cosine similarity as a metric, or the inner product metric is used on the normalized feature vectors to achieve retrieval equivalent to cosine similarity.
5. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The error sample library performs approximate nearest neighbor retrieval based on a vector index structure, which includes an inverted file and product quantization index structure and / or a hierarchical small-world graph index structure.
6. The online error sample elimination method based on self-supervised visual representation according to claim 5, characterized in that: The codebook and coarse quantizer of the vector index structure are trained offline and solidified using representative samples.
7. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The threshold strategy includes at least one of a global threshold strategy and a subdomain adaptive threshold strategy; the subdomain adaptive threshold strategy sets different thresholds according to category, camera position and / or time period.
8. The online error sample elimination method based on self-supervised visual representation according to claim 7, characterized in that: The threshold strategy periodically re-evaluates the threshold based on runtime similarity distribution and cost function.
9. The online error sample elimination method based on self-supervised visual representation according to claim 1, characterized in that: The metadata includes at least one of the following: timestamp, scene identifier, inference threshold, category information, and source camera information.
10. An online error sample elimination system based on self-supervised visual representation, characterized in that, include: The inference module is used to call the main visual model to perform inference on the input image or video frame and output the candidate target result; The feature extraction module is used to crop or extract the candidate regions corresponding to the candidate target results, and generate candidate feature vectors through a self-supervised visual feature encoding model. The error sample library module is used to store the feature vectors of error samples and their metadata, and supports online incremental writing; The similarity verification module is used to perform approximate nearest neighbor retrieval in the error sample library module to obtain similarity results, and to determine whether to mark the candidate target results as false alarms and filter them according to the threshold strategy; The false alarm entry module is used to generate a false alarm feature vector and write it along with metadata into the error sample database module after the candidate target result is confirmed as a false alarm by manual review.