Automatic data labeling method based on multi-model consistency verification

Abstract

The invention belongs to the technical field of machine learning, and discloses an automatic data labeling method based on multi-model consistency verification. According to the method, main angle analysis is carried out on feature subspaces of different network architecture pre-training models, a model homogeneity adjacency graph is constructed, the decision independence weight of each model is calculated, and collusion suspected samples are identified through prediction consistency and response region overlapping analysis. For a suspected sample, applying directional disturbance along the main direction of a shared feature subspace, and verifying a collusion effect by calculating disturbance sensitivity divergence between models; for non-collusion samples, decision independence weighted voting is adopted to generate annotations. According to the method, global labeling consistency verification is fused, and potential wrong labeling samples are identified. By establishing a multi-level consistency evaluation mechanism from static prediction to dynamic verification, collusion judgment of real consensus and surface consistency based on independent evidence can be distinguished, the reliability and accuracy of automatic labeling are remarkably improved, and the data labeling cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically, to a method for automatic data labeling based on multi-model consistency verification. Background Technology

[0002] With the rapid development of deep learning technology, automatic data annotation methods have become a key technology for solving the bottleneck of large-scale dataset construction. Among them, multi-model ensemble annotation is widely used because it can theoretically balance the decisions of multiple parties. However, existing automatic data annotation technologies face a challenge: the "collusion" error caused by model homogenization. In practical applications, when multiple deep neural networks with different architectures (such as convolutional networks, transformer networks, etc.) are deployed for collaborative annotation, these seemingly independent models often exhibit abnormally consistent judgments on specific samples, causing the annotation system to incorrectly classify these highly consistent results as high-confidence labels. The essence of this phenomenon is the hidden homogenization between models—despite different architectures, they share similar pre-training data sources, feature extraction patterns, and inductive biases, forming a false "independent multi-party confirmation." In sentiment analysis annotation in natural language processing, multiple classifiers may simultaneously misclassify ambiguous expressions; in visual object detection scenarios, detectors with different structures may consistently produce localization biases for objects under specific lighting conditions. This neglect of the inherent dependencies between models in the annotation system leads to the amplification and accumulation of systematic biases in the training data, seriously affecting the generalization performance of downstream applications. Existing automatic annotation methods lack effective mechanisms to detect and prevent this collusion phenomenon of "superficial majority, actual consensus," making it difficult to distinguish between truly reliable annotations based on multiple independent evidence and pseudo-high-confidence annotations based on common weaknesses.

[0003] In view of this, the present invention proposes an automatic data annotation method based on multi-model consistency verification to solve the above problems. Summary of the Invention

[0004] To overcome the aforementioned shortcomings of the prior art and to achieve the above objectives, the present invention provides the following technical solution: an automatic data annotation method based on multi-model consistency verification, comprising: Obtain the dataset to be labeled and K pre-trained classification models with different network architectures. Extract the intermediate layer feature representations of each pre-trained classification model on the shared probe dataset to form the feature representation matrix of each model. Perform subspace principal angle analysis on the feature representation matrices of every two models, calculate the overlap of paired feature subspaces, and construct a model homogeneity adjacency graph; Based on the homogeneity adjacency graph of the models, the decision independence weight of each model is calculated, and the model with higher homogeneity connectivity is assigned a lower decision independence weight. For each unlabeled sample in the labeled dataset, collect the prediction outputs of all K models and extract the class-sensitive spatial response map of each model to construct a sample-by-sample prediction-response composite descriptor; The prediction consistency score and response region overlap ratio are calculated from the prediction-response composite descriptor. Samples with prediction consistency scores exceeding a preset first threshold and response region overlap ratios exceeding a preset second threshold are identified as suspected collusion samples. For each suspected collusion sample, apply directional feature perturbation along the main direction of the shared feature subspace, and collect the prediction change vector of all models after perturbation; The perturbation sensitivity divergence between models is calculated based on the predicted change vector; when the perturbation sensitivity divergence is lower than the divergence threshold, the label confidence level of the sample is reduced and it is marked as awaiting manual review; when the perturbation sensitivity divergence is higher than the divergence threshold, the majority predictions before the perturbation are used as the labeling result. For samples that are not suspected of being accomplices, a weighted voting method based on decision independence is used to generate labels; All labeled data and confidence levels are aggregated, and a global label consistency check is performed on samples with the same labeled data to output the final labeled dataset.

[0005] The technical effects and advantages of the automatic data annotation method based on multi-model consistency verification in this invention are as follows: This invention significantly improves the reliability and accuracy of automatic annotation systems. It accurately distinguishes between genuine consensus based on independent evidence and superficial consistency based on common weaknesses, preventing the spread and accumulation of systematic annotation errors. Through the construction and analysis of model homogeneity adjacency graphs, it achieves optimal weight allocation based on model independence, overcoming the shortcomings of traditional majority voting where highly homogeneous model clusters dominate decision-making, ensuring a balance between decision diversity and representativeness. The directional feature perturbation mechanism provides the annotation system with proactive verification capabilities, revealing model sensitivity patterns that static prediction cannot detect, providing dynamic and in-depth evidence for annotation reliability assessment. The global annotation consistency verification mechanism automatically identifies potential outliers, ensuring the coherence of the dataset's internal structure and the rationality of label distribution. Response region analysis technology enables annotation decisions to no longer rely solely on output results, but to deeply understand the image regions the model focuses on, achieving an upgrade from "result consistency" to "process consistency" in evaluation. The multi-dimensional annotation reliability assessment system accurately identifies boundary samples requiring manual intervention, optimizing human resource allocation and significantly improving annotation efficiency while maintaining high-quality standards. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of the automatic data annotation method based on multi-model consistency verification according to the present invention. Detailed Implementation

[0007] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0008] This application provides an automatic data annotation method based on multi-model consistency verification. The execution entities of the method include, but are not limited to, automatic annotation systems, data analysis platforms, AI training assistance tools, and multi-model collaborative inference frameworks, which can be regarded as general computing nodes of this application. The annotation system includes, but is not limited to, at least one of cloud annotation engines, distributed annotation management systems, and consistency verifiers.

[0009] Please see Figure 1 In this embodiment of the invention, the automatic data annotation method based on multi-model consistency verification includes: We acquire the dataset to be labeled and K pre-trained classification models with different network architectures. We extract the intermediate layer feature representations of each pre-trained classification model on a shared probe dataset, forming the feature representation matrix for each model. The shared probe dataset is a set of samples used to probe the behavioral characteristics of the models. By observing the feature representations of different models on these samples, we can analyze the intrinsic relationships between the models. The pre-trained classification models include deep neural networks with different architectures, such as variants of ResNet, VGG, MobileNet, and Transformer. These models are pre-trained on the same domain data but have different structural characteristics. Intermediate layer feature representations are high-dimensional features that represent the data within the model. They are typically selected from the feature maps or vectors before the final fully connected layer of the model. These features contain the model's abstract understanding and discriminative information about the input data. The feature representation matrix records the feature response of each model on the probe dataset, providing basic data for subsequent subspace analysis.

[0010] Subspace principal angle analysis is performed on the feature representation matrices of every two models to calculate the overlap of paired feature subspaces and construct a model homogeneity adjacency graph. Subspace principal angle analysis is an effective technique for measuring the similarity between two high-dimensional feature spaces. By calculating the angles between principal vectors, the degree of overlap of feature subspaces is quantified. The analysis process first performs singular value decomposition on the feature representation matrices to retain the main feature directions; then, the angles between these principal vectors are calculated to form a principal angle sequence; finally, the statistics of the principal angles are taken as the subspace similarity index. The higher the feature subspace overlap, the more likely the two models are operating in similar feature spaces, and there may be knowledge or decision dependencies. The model homogeneity adjacency graph uses models as nodes and subspace overlap as edge weights, intuitively showing the similarity network between models and providing a graph theory perspective for identifying model clusters and independent models.

[0011] Based on a homogeneous adjacency graph, the decision independence weights of each model are calculated, with models having higher homogeneity connectivity receiving lower decision independence weights. Decision independence weights are a key indicator for evaluating a model's contribution to ensemble decision-making, reflecting its ability to provide independent judgments. The calculation process first calculates the connection strength of each model node in the homogeneous adjacency graph, quantifying the similarity between the model and other models; then, the reciprocal of the connection strength is taken to obtain the original independence index; finally, normalization is applied to form the final decision weight allocation. The weight allocation follows the principle of "higher independence, greater weight," ensuring that models providing unique perspectives receive greater decision-making power in ensemble decision-making, effectively reducing the risk of systemic errors caused by model collusion.

[0012] For each unlabeled sample in the unlabeled dataset, the prediction outputs of all K models are collected, and the class-sensitive spatial response map of each model is extracted to construct a sample-by-sample prediction-response composite descriptor. The prediction output includes the model's probability distribution vector and predicted class, reflecting the model's decision outcome. The class-sensitive spatial response map visualizes the image region of interest to the model using gradient weighting techniques, reflecting the region upon which the model's decision is based. These two types of information describe the model's decision-making behavior from both the outcome and process perspectives, respectively. Combined, they form the prediction-response composite descriptor, comprehensively characterizing the model's processing features of the samples. The descriptor construction employs a standardized feature extraction process to ensure the comparability of descriptors from different models, providing a foundational data structure for subsequent consistency analysis.

[0013] The prediction consistency score and response region overlap ratio are calculated from the prediction-response composite descriptor. Samples with both a prediction consistency score exceeding a preset first threshold and a response region overlap ratio exceeding a preset second threshold are identified as suspected collusion samples. The prediction consistency score measures the consistency of predictions from multiple models and is calculated as the proportion of models predicting the same category. The response region overlap ratio quantifies the degree of overlap of the models' regions of interest and is derived by calculating the spatial intersection-union ratio of the response map. When both metrics exceed the thresholds simultaneously, it indicates that multiple models not only provide the same prediction results but also make decisions based on similar image regions. This high consistency may stem from genuine model consensus or it may be a manifestation of model collusion. Suspected collusion samples require further perturbation verification to distinguish between genuine model consensus and false collusive consistency.

[0014] For each suspected collusion sample, a directional feature perturbation is applied along the principal direction of the shared feature subspace, and the predicted change vectors of all models are collected after the perturbation. Directional feature perturbation is an active probing method to verify the robustness of model decisions. By adding small changes in a specific direction, it tests the sensitivity and consistency of model decisions. Perturbation generation first determines the principal direction of the shared feature subspace among models, representing the feature dimensions that the models are jointly concerned with; then, a perturbation vector of fixed amplitude is generated along this direction; finally, the perturbation vector is applied to the original sample to generate the perturbation sample. The predicted change vector records the difference in model output before and after the perturbation, reflecting the model's sensitivity to the perturbation. In true model consensus, different models should respond differently to perturbations, exhibiting diversity in predicted changes; however, in the case of model collusion, multiple models will exhibit highly similar change patterns, reflecting the substantial similarity of their decision-making mechanisms.

[0015] The perturbation sensitivity divergence among models is calculated based on the predicted change vector. When the perturbation sensitivity divergence is below the divergence threshold, the labeling confidence level of the sample is reduced and it is marked as awaiting manual review. When the perturbation sensitivity divergence is above the divergence threshold, the majority prediction before the perturbation is used as the labeling result. Perturbation sensitivity divergence is a statistical indicator that quantifies the diversity of model responses. It assesses the consistency of the model population's response by calculating the coefficient of variation of the perturbation sensitivity. The calculation process first calculates the perturbation sensitivity of each model, i.e., the norm of the predicted change vector; then, it calculates the mean and standard deviation of the sensitivity of all models; finally, it calculates the ratio of the standard deviation to the mean to obtain the divergence index. The lower the divergence value, the more consistent the model population's response to the perturbation, potentially indicating model collusion; the higher the divergence value, the more diverse the model responses, reflecting true independent decision-making. Based on the divergence value, the system automatically determines the labeling strategy: a conservative strategy is adopted for low-divergence samples, marking them as awaiting review; for high-divergence samples, the majority voting result is used to form reliable labels.

[0016] For non-collusion suspect samples, a weighted voting system based on decision independence weights is used to generate labels. Weighted voting is a core mechanism for improving labeling quality by leveraging model diversity. By assigning greater weight to models with high independence, the accuracy of ensemble decision-making is optimized. The voting process first collects the predicted categories from each model, forming a candidate category set. Then, based on the decision independence weights of each model, the weighted vote for each candidate category is calculated. Finally, the category with the highest number of votes is selected as the labeling result. This weighted mechanism effectively overcomes the systematic bias that may be caused by the "majority rule" in traditional majority voting, especially when the judgments of a few independent models are actually more accurate. By adjusting the weights, the correct result can still be obtained. The weighted voting strategy provides a reliable automatic labeling mechanism for non-collusion samples, significantly improving labeling efficiency.

[0017] All labeled tags and confidence levels are aggregated, and a global label consistency check is performed on samples with the same labeled tags, outputting the final labeled dataset. Global label consistency check is the last line of defense to ensure the internal consistency of the labeling results. By analyzing the feature distribution of similar samples, potential mislabels are identified. The check process first groups the samples by label; then calculates the feature clustering and dispersion of samples within each group; finally, outliers that are significantly different from the main samples in the group are identified and marked as suspicious samples. This global consistency analysis can capture anomalies overlooked in local decision-making, further improving labeling quality. The final labeled dataset integrates all labeling results, including labels, confidence levels, and verification labels, providing a high-quality labeled data foundation for subsequent model training and validation.

[0018] In this embodiment of the invention, the detailed implementation steps for performing subspace principal angle analysis on the feature representation matrices of every two models, calculating the overlap of paired feature subspaces, and constructing a model homogeneity adjacency graph include: Singular value decomposition (SVD) is performed on the feature representation matrix of each model, retaining the first r principal singular vectors to form the feature subspace basis matrix of that model. SVD is an effective method for extracting the main structure of the feature matrix, capable of decomposing high-dimensional feature representations into orthogonal basis vectors and their importance weights. The decomposition process applies to the feature representation matrix... Applying the SVD algorithm, we obtain The decomposition form, in which and Contains left and right singular vectors. It includes singular values. The selection of the principal singular vectors is based on the magnitude of the singular values. The left singular vectors corresponding to the largest r singular values ​​are retained to form the basis matrix of the feature subspace. The parameter r is usually determined by the proportion of cumulative explained variance. The smallest r value that can explain more than 90% of the original feature variance is selected to ensure that the basis matrix contains the main information dimensions of the model feature space, while effectively reducing the complexity of subsequent calculations.

[0019] For any two models, calculate all principal angles between their feature subspace basis matrices and the basis vectors. Principal angles are a geometric tool for quantifying the similarity between two subspaces, representing the optimal representation of a basis vector from one subspace in the other. The calculation process first standardizes the two feature subspace basis matrices to ensure all basis vectors are unit vectors; then, it constructs a special matrix product QR decomposition problem to solve for the canonical correlation between the two basis vectors; finally, it obtains the principal angle sequence by calculating the inverse cosine of the canonical correlation. The number of principal angles calculated is equal to the smaller of the two subspace dimensions. Each principal angle reflects the degree of difference between the two subspaces in a given dimension; the smaller the angle, the more similar the two subspaces are in that dimension. The mathematical form of the principal angles can be expressed as: ; in, It is the first From a personal perspective It is a matrix The A singular value, and These are the feature subspace basis matrices of the two models.

[0020] The average of the cosine values ​​of all principal angles is taken as the feature subspace overlap of the model pair. Overlap is a single indicator that integrates information from all principal angles, intuitively reflecting the overall similarity between two feature subspaces. The calculation process first takes the arithmetic mean of the cosine values ​​of the principal angles to obtain the average projection similarity of the two subspaces in each dimension. This mean reflects the average projection efficiency from one subspace to another. The overlap value ranges from [0,1]. The closer the value is to 1, the more similar the two subspaces are; the closer the value is to 0, the more orthogonal and completely unrelated the two subspaces are. The formula for calculating overlap is: ; in, It is a model and The feature subspace overlap, and It is the number of principal singular vectors retained by the two models. It is the first From a personal perspective.

[0021] A homogeneous adjacency graph is constructed using all K models as nodes and the overlap of the feature subspaces of each model pair as edge weights. The homogeneous adjacency graph is a network structure describing the similarity relationships between models, intuitively displaying similar clusters and independent individuals within a model group. The construction process first creates a graph structure containing K nodes, each node corresponding to a pre-trained model; then, based on the overlap of the feature subspaces of each pair of models, weighted edges are created between the corresponding nodes, with the edge weight representing the overlap; finally, a fully connected weighted undirected graph is formed, comprehensively recording the pairwise similarity relationships between models. The homogeneous adjacency graph not only records the direct similarity relationships between models but also reveals implicit structures within the model group through graph algorithms, such as similar clusters, central models, and bridging models, providing a graph theory foundation for subsequent decision independence analysis and collusion detection.

[0022] Edges with weights below the overlap threshold in the homogeneous adjacency graph of the models are pruned, retaining edges indicating significant homogeneous associations. Graph pruning is a key step in simplifying model relationship networks, highlighting strong association patterns between models by removing weak similarity connections. The pruning process first determines an appropriate overlap threshold, usually by analyzing the statistical characteristics of the overlap distribution or domain knowledge; then, it examines the weight of each edge in the graph and removes edges below the threshold; finally, a sparsified homogeneous adjacency graph is obtained, retaining only significantly similar model pairs. The pruned graph structure more clearly shows high-similarity clusters in the model group, facilitating the identification of potential colluding model groups, while reducing the complexity of subsequent computations. This threshold-based graph sparsification method is a common technique in graph analysis, effectively filtering noisy connections while preserving key structural information.

[0023] In this embodiment of the invention, the detailed implementation steps for calculating the decision independence weights of each model based on a homogeneous adjacency graph include: The weighted degree value of each model node in the homogeneous adjacency graph is calculated. The weighted degree is the sum of the weights of all edges connected to that node. The weighted degree is a fundamental indicator quantifying the connectivity strength of a model in a homogeneous network, reflecting the overall similarity between the model and other models. The calculation process directly sums the weights of all edges connected to the node to obtain a single numerical index. The mathematical expression is: ; in, It is a model The weighted degree value, It is a model With model The weights of the edges between them (the overlap of the feature subspaces). This represents the total number of models. Models with higher weighting values ​​are highly similar to many other models and may be located at the center of similarity clusters, exhibiting lower independence. Conversely, models with lower weighting values ​​have lower similarity to other models, possess more unique decision-making characteristics, and are more independent. The weighting value provides an intuitive measure for the initial assessment of model independence and forms the basis for subsequent independence weight calculations.

[0024] The original independence index for each model is obtained by taking the reciprocal of the weighted value of each model node. This original independence index converts the similarity metric into an independence metric by inverting the weighted values, adhering to the design principle of "lower similarity, higher independence." The calculation process uses a simple reciprocal transformation to map the weighted values ​​to the independence space. The expression is: ; in, It is a model The original independence index, That is the corresponding weighted degree value. It is a small positive number (usually set to 1e-5) to prevent division by zero. The reciprocal transformation ensures that models highly similar to other models receive low independence scores, while models significantly different from other models receive high independence scores. This transformation preserves the order of the original similarity relationship while adjusting the numerical distribution, making it more suitable for expressing the concept of independence and providing a suitable numerical basis for subsequent weight normalization.

[0025] Normalization is performed on the original independence indices of all models to obtain the decision independence weights for each model. Normalization is a crucial step in converting the original independence indices into standardized weights, ensuring a reasonable weight distribution and a sum of 1. The process uses a sum-of-the-parts normalization method, dividing the original independence index of each model by the sum of all model indices to obtain a standardized weight distribution. The calculation formula is: ; in, It is a model The weight of decision independence, It is its original independence indicator. This represents the total number of models. The normalized weights are distributed in the interval [0,1] and satisfy... This approach aligns with probability distribution characteristics and is suitable for subsequent weighted voting processes. This graph-based weight allocation scheme effectively transforms the structural relationships between models into decision weights, enabling highly independent models to gain greater influence in ensemble decision-making, thereby improving the overall accuracy and robustness of the system.

[0026] When the decision independence weight of a model falls below the minimum weight threshold, that model is excluded from the voting set of the current labeled round. Model exclusion is a protective mechanism to prevent low-independence models from interfering with decision quality. By setting a weight lower limit, highly similar redundant models are filtered out. The exclusion process first determines an appropriate minimum weight threshold, typically the theoretical average weight (…). A certain proportion (e.g.) Then, the independence weights of each model are checked, and models below a threshold are marked as exclusions. Finally, the weight distribution of the remaining models is adjusted to ensure that the total weight remains 1. This model selection mechanism avoids duplicate voting by highly similar models, reduces the risk of decision bias, and enhances the diversity and independence of the voting set. Model exclusion is particularly suitable for handling heterogeneous sets containing multiple similar derived models, effectively balancing the decision influence among model families and laying a fairer foundation for subsequent weighted voting.

[0027] In this embodiment of the invention, the detailed implementation steps for collecting the prediction outputs of all K models and extracting the category-sensitive spatial response map of each model to construct a sample-by-sample prediction-response composite descriptor include: Unlabeled samples are input into each model to obtain its probability vector and predicted class label. The probability vector is the raw probability distribution output by the model, containing the model's confidence assessment for each class; the predicted class label is the index of the class with the highest probability, representing the model's final decision. The acquisition process uses a standard forward inference flow: samples are input into the model, the activation values ​​of the output layer are obtained through the model's forward propagation, and the probability distribution is obtained after softmax transformation. For each model... Its output can be represented as: ; ; in, It is a model For the sample The probability vector, These are the activation values ​​of the model's output layer. These are the predicted category identifiers. This basic prediction information directly reflects the model's decision results and is the first component in constructing the prediction-response descriptor, providing foundational data at the result level for subsequent consistency analysis.

[0028] For each model, the gradient of the output score corresponding to the predicted class label with respect to the feature map of the last convolutional layer is calculated. This gradient is then used to perform channel-weighted summation of the feature map, generating the class-sensitive spatial response map for that model. The class-sensitive spatial response map is an important tool for visualizing the model's decision-making process, highlighting spatial regions that significantly contribute to the prediction results through gradient weighting. The generation process is based on the Grad-CAM method. First, the gradient of the predicted class score with respect to the feature map is calculated. Then, the gradient is averaged across the spatial dimension to obtain the importance weight of each channel. Finally, a weighted summation and ReLU activation are used to generate the final response map. The mathematical expression is: ; ; in, It is a category For the feature map The importance weight of each channel It is a category The predicted score, These are the activation values ​​of the feature map. This is the final category-sensitive response map. The response map visually displays the image regions that the model focuses on, reflects the visual basis of the model's decisions, and provides important process-level information for understanding model behavior and detecting collusion.

[0029] Binarization is performed on the category-sensitive spatial response map to obtain a mask of the response activation region. Binarization is a simplified step that converts a continuous response map into a discrete mask, facilitating subsequent region comparison and overlap analysis. The process first normalizes the response map, mapping the values ​​to the [0,1] interval; then, an appropriate activation threshold is set (usually 0.5 or adaptively determined using the Otsu method); finally, binarization is performed based on the threshold to generate a binary mask representing the key regions. The binarized mask clearly identifies the key and secondary regions in the model's decision-making, simplifying the complexity of region overlap analysis while preserving the core spatial information of the response map. This region simplification method is a common technique in computer vision, effectively extracting salient regions from images and providing a unified region representation for subsequent overlap analysis.

[0030] The probability vector, predicted class identifier, and response activation region mask of each model are combined to form a composite prediction-response descriptor for that model on the sample. The composite descriptor is a unified data structure integrating prediction results and decision-making processes, comprehensively describing the model's processing characteristics of the sample. The combination process creates multi-field structured data, including probability distributions and class identifiers at the prediction level, and response region masks at the process level, forming a complete description of model behavior. The structural design follows a balance between information integrity and computational efficiency, including key decision information while avoiding the storage and processing of redundant data. This description method, combining results and processes, contains richer information than simple prediction results, enabling a deeper analysis of model behavior and interrelationships, and providing a multi-dimensional information foundation for subsequent consistency analysis and collusion detection.

[0031] In this embodiment of the invention, the detailed implementation steps for identifying samples whose prediction consistency score exceeds a preset first threshold and whose response region overlap ratio simultaneously exceeds a preset second threshold as suspected collusion samples include: The prediction consistency score is obtained by counting the number of models with the same predicted class label among all K models and dividing the number by K. The prediction consistency score is a fundamental indicator quantifying the degree of consensus among multiple independent models, reflecting the level of consensus reached. The calculation process first determines the majority class predicted by each model, and then calculates the proportion of models supporting that class. The score ranges from [value missing]. ,in This represents the total number of categories. A score closer to 1 indicates greater consistency in predictions between models. This indicates that the model's predictions are increasingly divergent. The consistency score is calculated using the following formula: ; in, It is the prediction consistency score. It is a model Prediction categories, It is the most frequently occurring prediction category. This refers to the total number of models. High predictive consistency is generally a positive sign, indicating that the samples have clear class characteristics; however, abnormally high consistency may also be a sign of model collusion, requiring a comprehensive judgment in conjunction with other indicators.

[0032] The cross-union ratio (IoU) of each pair of response activation region masks for all models is calculated, and the average IoU of all model pairs is taken as the response region overlap ratio. The response region overlap ratio is a spatial measure for evaluating the similarity of model decision-making processes. By analyzing the degree of overlap in the models' areas of interest, it reveals the similarity of the decision-making processes. The calculation process first calculates the IoU for each pair of model response masks to quantify the degree of overlap between the two regions; then, the average IoU of all model pairs is taken to obtain the overall overlap ratio index. The formula for calculating the IoU is: ; ; in, It is a model and The response mask crossover ratio, It is the overall overlap ratio of the response area. This represents the total number of models. The overlap ratio ranges from [0,1]. A larger value indicates that the models focus on similar regions, and the potential risk of collusion is higher. In particular, when multiple structurally different models focus on the exact same region, it often indicates the existence of shortcut features or common inductive bias in the data.

[0033] A sample is considered a suspected collusion sample when both the prediction consistency score and the response region overlap ratio are greater than a second threshold. The determination of suspected collusion samples is based on a comprehensive judgment of both prediction and response consistency, identifying samples that may pose a risk of model collusion. The determination process sets two independent thresholds: the first threshold targets prediction consistency and is typically set to a high value (e.g., 0.8-0.9); the second threshold targets the response overlap ratio and is typically set to a medium value (e.g., 0.6-0.7). When both conditions are met simultaneously, it indicates that different models not only provide highly consistent prediction results, but these results are also based on highly similar decision regions. This dual consistency anomaly is a strong signal of potential collusion and requires further verification. The collusion determination uses an "AND" logic, requiring both conditions to be true simultaneously. This conservative strategy reduces the false positive rate and ensures that labeled suspected collusion samples possess sufficient suspicious characteristics.

[0034] When the prediction consistency score is greater than the first threshold but the response region overlap ratio is less than or equal to the second threshold, the multi-model predictions are determined to be independent and consistent, and the sample is marked as a high-confidence labeled sample. High-confidence labeled samples represent the ideal situation of consistent predictions but independent decision-making processes, indicating that different models have reached the same judgment result through different feature paths. The judgment process examines the combined state of prediction consistency and response overlap. When predictions are highly consistent but response regions differ significantly, it indicates that the models have verified the same conclusion from different perspectives. This multi-path verification greatly enhances the reliability of the judgment. High-confidence labeling reflects the scientific ideal state of "multiple pieces of evidence supporting the same conclusion," providing high-quality deterministic samples for automatic labeling. This judgment method based on process differences and result consistency effectively distinguishes between true model consensus and superficial result consistency, providing a reliable confidence assessment for subsequent labeling decisions.

[0035] In this embodiment of the invention, the detailed implementation steps of applying a directional feature perturbation to each suspected collusion sample along the main direction of the shared feature subspace and collecting the predicted change vectors of all models after the perturbation include: From the homogeneous adjacency graph of the models, the model pair with the largest edge weight is selected, and the shared feature subspace basis matrix of the model pair is extracted. The shared feature subspace basis matrix represents the common feature directions of highly similar model pairs and is the key foundation for generating effective perturbations. The extraction process first identifies the edge with the largest weight in the homogeneous adjacency graph, corresponding to the model pair with the highest similarity; then, combining the feature subspace basis matrices of the two models, the main shared feature directions are extracted through joint singular value decomposition. These shared directions are feature dimensions that both models are simultaneously sensitive to, and perturbations along these directions are most likely to trigger the common response of the models, making them suitable for collusion verification. The most similar model pair is selected as a reference point to ensure that the generated perturbations can effectively detect collusion relationships between models, providing ideal perturbation directions for subsequent sensitivity analysis.

[0036] The input features of suspected collusion samples are projected onto the first principal direction of the shared feature subspace basis matrix to obtain the projected components. These projected components represent the feature intensity of the sample along the shared principal direction, reflecting the sample's performance in the most sensitive feature dimension. The calculation process first involves performing an inner product operation between the sample features and the first principal vector of the shared basis matrix to obtain the scalar projection of the sample along that direction. This projected value quantifies the feature intensity of the sample along the shared sensitive direction, providing a baseline value for perturbation generation. The first principal direction typically corresponds to the largest shared singular value, representing the model's most commonly sensitive feature dimension; perturbations along this direction are most likely to trigger significant and relevant changes in the model response. The projection operation is a fundamental transformation in linear algebra, capable of accurately decomposing the feature components of the sample along a specific direction, providing a precise quantitative basis for subsequent directional perturbations.

[0037] A directional perturbation vector is generated by scaling the projected components with a preset perturbation step size. This directional perturbation vector represents a controlled change along a shared sensitivity direction, used to test the model's sensitivity to changes in specific feature dimensions. The generation process first sets an appropriate perturbation step size, typically a small fraction of the original feature amplitude (e.g., 0.01-0.05); then, this step size is combined with the projected components to calculate the perturbation amplitude along the principal direction; finally, a complete perturbation vector is constructed using the first principal vector as the direction and the calculated amplitude as the magnitude. The choice of perturbation step size balances detectability and minimization: it is small enough to ensure the perturbation does not alter the true semantics of the sample, while being large enough to trigger a measurable change in the model response. This projection-based perturbation generation method ensures the accuracy of the perturbation direction and the reasonableness of its amplitude, providing a controllable and effective input variation for model sensitivity testing.

[0038] Perturbation samples are generated by superimposing directional perturbation vectors onto the original input of a suspected collusion sample. These perturbation samples are subtle variants of the original samples, preserving the original semantics while introducing directional feature changes. The generation process directly adds the perturbation vector to the features of the original sample, forming a new input sample. For high-dimensional inputs such as images, the perturbation vector first needs to be mapped back from the feature space to the original input space to ensure the perturbation is applied at the correct dimension and scale. The perturbated sample is visually almost indistinguishable from the original sample, but exhibits subtle and precise changes in specific feature dimensions. These changes are sufficient to trigger a sensitive response from the model without altering the true class of the sample. Perturbation sample generation is a gentle application of adversarial example techniques; it does not aim to mislead the model but rather to probe the sensitivity and robustness of the model's decisions, providing a valuable proactive detection tool for understanding model behavior.

[0039] The perturbation sample is input into all K models, and the difference between the probability vectors of each model before and after the perturbation is calculated, denoted as the predicted change vector for each model. The predicted change vector records the model's sensitive response to the perturbation and is key data for analyzing the robustness of model behavior. The calculation process first inputs the perturbation sample into each model to obtain the probability vector after the perturbation; then, the difference is calculated between this vector and the original probability vector before the perturbation to obtain the vector representing the change. This change vector includes the increase or decrease in the probability of each class, comprehensively recording the impact pattern of the perturbation on the model's decision-making. Under independent model decision-making, the change vectors of different models should exhibit diverse patterns; however, under collusion, multiple models will produce highly similar change vectors, reflecting that they share similar decision-making mechanisms and feature sensitivities. This behavior analysis method based on perturbation response provides a dynamic perspective beyond static prediction, capable of deeply revealing the essential similarities and differences in model decision-making mechanisms, and providing strong behavioral evidence for collusion testing.

[0040] In this embodiment of the invention, the detailed implementation steps for calculating the perturbation sensitivity divergence between models based on the predicted change vector include: Calculate the L2 norm of the predicted change vector for each model, denoted as the model's perturbation sensitivity. Perturbation sensitivity is a scalar measure that quantifies the strength of a model's response to a perturbation, reflecting how sensitive the model's decisions are to a specific perturbation. The calculation process uses the standard L2 norm (Euclidean norm) to compress the multidimensional predicted change vector into a single sensitivity value. The calculation formula is: ; in, It is a model The sensitivity to disturbances It is its predicted change vector. It is the first The probability changes of each category This represents the total number of categories. A higher sensitivity value indicates a stronger response to disturbances and more unstable decision-making; a lower value indicates stronger resistance to disturbances and more stable decision-making. This norm-based sensitivity calculation method comprehensively considers changes in the probabilities of all categories, providing a comprehensive and accurate measure of sensitivity and offering a comparable standardized indicator for subsequent statistical analysis.

[0041] Calculate the mean and standard deviation of the perturbation sensitivity for all K models. The mean and standard deviation are fundamental statistics describing the sensitivity distribution, reflecting the average response level and the degree of difference in response among the model population to perturbations. The calculation process uses standard sample mean and sample standard deviation formulas to process the sensitivity values ​​of all models. The formula for calculating the mean is: ; The formula for calculating standard deviation is: ; in, It is the mean sensitivity. It is the standard deviation of sensitivity. It is a model The sensitivity to disturbances This represents the total number of models. The mean reflects the overall sensitivity level, while the standard deviation reflects the degree of difference in sensitivity among models. When models make independent decisions, different models typically exhibit diverse sensitivities to the same disturbance, leading to larger standard deviations; conversely, when models collude, multiple models will exhibit similar sensitivities, resulting in smaller standard deviations. These two statistics together constitute the basic description of the sensitivity distribution, providing the necessary statistical basis for calculating the divergence index.

[0042] Dividing the standard deviation by the mean yields the coefficient of variation for perturbation sensitivity, denoted as the perturbation sensitivity divergence. The perturbation sensitivity divergence is a standardized measure of dispersion; by eliminating the influence of the mean, it provides a pure assessment of the diversity of the model's response. The calculation uses the coefficient of variation formula, dividing the standard deviation by the mean to obtain the relative divergence index. The divergence calculation formula is: ; in, It is the perturbation sensitivity divergence. It is the standard deviation. It is the mean. It is a small positive number to prevent division by zero (usually set to 1e-5). The smaller the divergence value, the more consistent the model population's response to the disturbance, which may indicate collusion; the larger the value, the more diverse the response, reflecting independent decision-making characteristics. The coefficient of variation is a commonly used normalized discrete measure in statistics. It is not affected by the original dimensions and can fairly compare the differences in response under different disturbances, providing a standardized statistical basis for determining collusion.

[0043] When the perturbation sensitivity divergence is below the divergence threshold, it is determined that all models exhibit a synchronous response to perturbations in the shared bias direction, confirming the existence of a collusion effect. The collusion effect determination is based on a decision rule of statistical significance, using the divergence threshold to distinguish between normal response differences and anomalous consistency. The determination process first determines an appropriate divergence threshold, typically determined through extensive experiments or theoretical analysis, with a typical value of 0.2-0.3. Then, the calculated divergence value is compared with the threshold. When it is below the threshold, the model group is considered to have an anomalously consistent response to the perturbation, confirming the existence of a collusion effect. This anomalously consistent response pattern indicates that models with different architectures have captured the same feature patterns and formed similar decision-making mechanisms. This cross-architecture consistency exceeds the scope of normal independent learning, strongly suggesting the existence of shortcut features or common biases in the data, causing different models to form "coincidentally" erroneous decision-making patterns. The collusion effect indeed provides clear risk signals for subsequent annotation processing, guiding the system to adopt a more cautious annotation strategy.

[0044] In this embodiment of the invention, the detailed implementation steps for generating labeled samples by weighted voting using decision independence weights for non-conspiracy suspect samples include: The predicted category identifiers of each model for the non-collusion suspect sample are aggregated to compile a set of supporting models for each candidate category. This set of supporting models records all models supporting each candidate category and serves as the foundation for weighted voting. The aggregation process first identifies all occurring predicted categories, forming a set of candidate categories; then, for each candidate category, the indexes of all models predicting that category are collected, forming a set of models supporting that category. This set not only records the support level of the category but also retains detailed information on which specific models support that category, providing a complete decision-making basis for subsequent weighted calculations and analysis. Constructing the set of supporting models is a simple yet crucial data organization step, integrating scattered model predictions into structured category support information, facilitating subsequent weight application and vote calculation.

[0045] For each candidate category, the decision independence weights of each model in its supporting model set are summed to obtain the weighted vote value for that candidate category. The weighted vote value is an enhanced voting statistic that considers model independence, adjusting the weights to give independent models a greater influence. The calculation process iterates through the supporting model set for each candidate category, summing the independence weights of all models in the set to obtain the total vote value for that category. The mathematical expression is: ; in, It is a category The weighted vote value, Supported categories The set of models, It is a model The decision independence weighting mechanism ensures that the opinions of highly independent models receive greater attention, prevents highly similar model clusters from dominating decisions through numerical superiority, effectively balances the trade-off between "majority rule" and "independent thinking," and provides a fairer and more accurate decision-making mechanism for automatic labeling.

[0046] The candidate category with the highest weighted vote value is selected as the label for the sample. The selection of the category with the highest vote is the final decision step in the voting process, determining the final labeling result by comparing the votes for each category. The selection process directly compares the weighted vote values ​​of all candidate categories and selects the category with the highest number of votes as the label. When multiple categories have the same number of votes, auxiliary rules can be used, such as selecting the category with higher model confidence, or leaving it undecided for later verification. The maximum vote mechanism is a classic decision-making method in ensemble learning. By pooling the wisdom of multiple models, it reduces the bias and error of individual models, improving the accuracy and robustness of the overall decision. This labeling method based on weighted majority voting considers both the quantity of models and balances the differences in model quality, providing a theoretically sound and practically effective decision-making strategy for automatic labeling systems.

[0047] The confidence score of a label is calculated as the ratio of the highest weighted vote to the sum of the weighted votes for all candidate categories. Label confidence is a crucial indicator of label reliability, reflecting the degree of consensus among the model's members regarding the labeling results. The calculation involves dividing the highest vote by the sum of all votes to obtain a standardized confidence score. The confidence score calculation formula is as follows: ; in, It is to label the confidence level. It is the vote value of the category with the most votes. It is the set of all candidate categories. It is a category The number of votes received. The confidence level ranges from [value missing]. , where |C| is the number of candidate categories. A value closer to 1 indicates more certain labeling and greater consistency among most models; a value closer to 1 indicates less certain labeling and greater divergence among models. Confidence level provides a quantitative basis for subsequent labeling quality assessment and selection, enabling the system to distinguish between high-confidence and low-confidence labeling results, and providing an important reference for quality control and prioritization of manual review.

[0048] In this embodiment of the invention, the detailed implementation steps for performing a global annotation consistency check on samples with the same annotation labels and outputting the final labeled dataset include: All labeled samples are grouped according to their labels. For each group, the mean vector of the intermediate layer feature representations of each model is extracted, and the intra-group feature dispersion is calculated. Intra-group feature dispersion is a statistical indicator that evaluates the consistency of features among samples of the same class and is used to detect potential mislabeling. The calculation process first groups all samples labeled as belonging to the same category; then, for each sample within a group, the feature representations of the intermediate layers of the model are extracted and the mean vector is calculated as a feature summary of the sample; finally, the dispersion of the feature vectors of all samples within the group is calculated to quantify the variability of features within the group. Dispersion calculation typically uses the average pairwise distance or the trace of the feature covariance matrix, reflecting the degree of dispersion of feature distribution. Under correct labeling, samples of the same class usually have similar feature representations, exhibiting low dispersion; while mislabeling introduces anomalous samples, leading to increased dispersion. Feature consistency analysis from a global perspective can discover anomalies overlooked in local decision-making, providing an additional verification mechanism for labeling quality control.

[0049] When the dispersion of features within a group exceeds a dispersion threshold, hierarchical clustering is performed on the samples within that group. Samples whose feature distances from most samples within the group exceed an outlier distance threshold are marked as potential mislabeled samples. Mislabeled sample identification is a fine-grained process of discovering outliers within a group through cluster analysis, used to locate potential labeling errors. The identification process first determines the dispersion threshold, usually based on the dispersion distribution of normal groups. When the dispersion of a group exceeds the threshold, detailed cluster analysis is initiated, using a hierarchical clustering algorithm to group samples within the group according to feature similarity. Then, the dominant cluster and outlier clusters are identified, and the distance between the sample and the center of the dominant cluster is calculated. Finally, based on the distance threshold, samples with excessively large distances are marked as potential mislabeled samples. Hierarchical clustering is suitable for handling situations with an unknown number of clusters, naturally discovering hierarchical structures and outliers in the data, providing a flexible and effective analytical tool for mislabeling identification. This clustering-based mislabeling detection method, combining distance metrics and distribution analysis, can accurately locate outlier samples, providing specific correction targets for subsequent labeling corrections.

[0050] The confidence level of potentially mislabeled samples is downgraded to the "awaiting review" level. Downgrading the confidence level is a conservative strategy for handling suspicious labels, preventing the propagation of errors by reducing certainty. The downgrading process directly modifies the confidence level of potentially mislabeled samples, marking them as low-confidence samples requiring manual review, regardless of their original confidence level. This strategy ensures that the high-confidence labels output by the system have extremely high accuracy, while guiding suspicious samples to the manual review process, avoiding the accumulation of errors in automatic labeling. Confidence level systems are typically divided into multiple levels, such as high confidence, medium confidence, low confidence, and awaiting review. Different levels of labeling can be applied to different scenarios or undergo different levels of review, forming a tiered quality control system.

[0051] All annotation labels, confidence levels, and review markers are integrated to output the final labeled dataset. This final labeled dataset is the method's final output, integrating all annotation results and quality assessment information. The integration process organizes sample IDs, annotation labels, confidence levels, and review markers into a structured dataset, forming a complete annotation output. The final dataset not only includes the annotation results but also records the reliability assessment and processing recommendations for the annotations, providing comprehensive information support for subsequent data use and quality management. The data format design follows the principles of usability and completeness, facilitating machine processing while including human-understandable quality indicators, supporting seamless integration of automated processes and human intervention. This information-rich annotation output format surpasses the traditional simple output containing only labels, providing richer data quality references for downstream tasks and contributing to the construction of more reliable and transparent AI systems.

[0052] In this embodiment of the invention, the detailed implementation steps for selecting the model pair with the largest edge weight from the homogeneous adjacency graph and extracting the shared feature subspace basis matrix of the model pair include: In a homogeneous adjacency graph, all edges are traversed, and the two model nodes connected by the edge with the largest edge weight are selected as the model pair with the largest edge weight. The model pair with the largest edge weight represents the two most similar models in the feature subspace and serves as an ideal reference point for identifying shared feature directions. The selection process employs a full graph traversal strategy, examining every edge in the adjacency graph, recording and comparing its weights, and finally determining the edge with the largest weight. Specifically, linear traversal or a priority queue method can be used. For large-scale graphs with a large number of edges, a heap data structure can be used to optimize the maximum value search. When multiple edges have the same maximum weight, auxiliary rules can be used, such as selecting node pairs with higher connectivity, or directly randomly selecting one pair. This model pair selection strategy based on maximum similarity ensures the targetedness and effectiveness of subsequent perturbation direction extraction, providing the most sensitive testing foundation for collusion detection.

[0053] Extract the feature subspace basis matrices for each of the two models in the model pair. These basis matrices contain the main structural information of the model feature spaces and are the raw material for constructing the shared subspace. The extraction process directly obtains the basis matrices already calculated in the principal angle analysis stage of the subspace; these matrices contain the main orthogonal basis vectors of their respective model feature spaces. For each model in the model pair, its feature subspace basis matrix is ​​typically a matrix composed of the first r left singular vectors obtained by singular value decomposition of the feature representation matrix. These basis matrices capture the main directions of change of the model in the feature space and contain the core feature dimensions of the model's decisions. The dimensions of the two basis matrices may differ, depending on the number r of principal components retained by each model; this difference needs to be specially handled in the subsequent joint decomposition. Extracting the basis matrices of the two models separately is a necessary preparatory step for joint analysis, providing a complete feature space representation for discovering shared feature directions.

[0054] Joint singular value decomposition (SVD) is performed on the basis matrices of two eigenspaces to extract the top p joint singular vectors corresponding to the maximum singular value. SVD is a core technique for discovering shared structures between two eigenspaces; it identifies common feature directions by analyzing the correlation between the two basis matrices. The decomposition process first combines the two basis matrices through a product operation to form an incidence matrix. ,in and These are the eigenspace basis matrices of the two models; then, standard singular value decomposition is performed on the incidence matrix R to obtain... Finally, through matrix multiplication... and The original basis matrix is ​​projected onto the shared space, resulting in two sets of joint singular vectors. These joint singular vectors are sorted by their corresponding singular values; larger singular values ​​indicate a higher degree of sharing in that direction across the two feature spaces. The first p joint singular vectors correspond to the p largest singular values, representing the most significant shared directions in the two feature spaces. These directions are feature dimensions that both models are simultaneously sensitive to, and perturbations along these directions are most likely to trigger a collusive response in the models. The parameter p is typically set to a small value (e.g., 1-3) to ensure that the most important shared directions are captured while avoiding the introduction of too many secondary or noisy directions.

[0055] The matrix formed by the first p joint singular vectors is used as the basis matrix of the shared feature subspace. The shared feature subspace basis matrix is ​​a compact representation of the intersection of the feature spaces of the two models, containing the main feature directions of common interest. The construction process directly combines the first p joint singular vectors column-wise to form a matrix of dimension d×p, where d is the dimension of the original feature space and p is the number of selected shared vectors. This basis matrix can be represented as: ; in, It is the first The nth joint singular vector, corresponding to the nth... Large singular values. Each column of the basis matrix is ​​a unit vector representing an orthogonal direction in the feature space, arranged from highest to lowest shared importance. The shared feature subspace basis matrix provides precise directional guidance for subsequent directional perturbation generation, enabling perturbations to be applied along the shared dimensions most sensitive to the model, maximizing the likelihood of triggering potential collusive responses. This shared space extraction method based on joint decomposition captures the essential similarity between models more accurately than simple feature averaging or concatenation, providing a mathematically rigorous feature foundation for collusion detection.

[0056] This invention achieves high-quality automatic annotation of datasets through multi-model feature subspace analysis, homogeneous network construction, collusion behavior detection, and perturbation verification. The model collusion detection method of this invention can effectively identify multi-model consistency artifacts, providing annotation results with high confidence and offering a systematic solution for data annotation in the fields of machine learning and artificial intelligence.

[0057] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0058] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0059] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An automatic data annotation method based on multi-model consistency verification, characterized in that, include: Obtain the dataset to be labeled and K pre-trained classification models with different network architectures. Extract the intermediate layer feature representations of each pre-trained classification model on the shared probe dataset to form the feature representation matrix of each model. For each pair of models, perform subspace principal angle analysis on the feature representation matrix, calculate the overlap of the paired feature subspaces, and construct a model homogeneity adjacency graph; Based on the homogeneous adjacency graph of the models, the decision independence weights of each model are calculated; For each unlabeled sample in the dataset to be labeled, collect the prediction outputs of all K models and extract the class-sensitive spatial response map of each model to construct a sample-by-sample prediction-response composite descriptor; The prediction consistency score and response region overlap ratio are calculated from the prediction-response composite descriptor. Samples with prediction consistency scores exceeding a preset first threshold and response region overlap ratios exceeding a preset second threshold are identified as suspected collusion samples. For each suspected collusion sample, a directional feature perturbation is applied along the main direction of the shared feature subspace, and the prediction change vectors of all models after the perturbation are collected. The perturbation sensitivity divergence between models is calculated based on the predicted change vector; when the perturbation sensitivity divergence is lower than the divergence threshold, the labeling confidence level of the sample is reduced and it is marked as awaiting manual review; when the perturbation sensitivity divergence is higher than the divergence threshold, the majority prediction before the perturbation is used as the labeling result. For samples that are not suspected of being accomplices, a weighted vote is performed using the aforementioned decision independence weight to generate labeling tags; All labeled data and confidence levels are aggregated, and a global label consistency check is performed on samples with the same labeled data to output the final labeled dataset.

2. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The subspace principal angle analysis is performed on the feature representation matrices of every two models to calculate the overlap of paired feature subspaces and construct a model homogeneity adjacency graph, including: Singular value decomposition is performed on the feature representation matrix of each model, and the first r principal singular vectors are retained to form the feature subspace basis matrix of the model; For any two models, calculate all principal angles between the two sets of basis vectors in the feature subspace basis matrix; The average of the cosine values ​​of all the principal angles is taken as the feature subspace overlap of the model pair. Using all K models as nodes and the overlap of the feature subspace of each model pair as the edge weight, construct the homogeneity adjacency graph of the models; Edges with weights below the lower limit of overlap in the homogeneous adjacency graph of the model are pruned, retaining edges that indicate significant homogeneous associations.

3. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The step of calculating the decision independence weights of each model based on the homogeneous adjacency graph of the models includes: Calculate the weighted degree value of each model node in the homogeneous adjacency graph of the model, where the weighted degree value is the sum of the weights of all edges connected to that node; Take the reciprocal of the weighted value of each model node to obtain the original independence index of each model; Normalization is performed on the original independence indices of all models to obtain the decision independence weights of each model; When the decision independence weight of a model is lower than the minimum weight threshold, the model is excluded from the voting set of the current annotation round.

4. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The process of collecting the prediction outputs of all K models and extracting the category-sensitive spatial response map of each model to construct a sample-by-sample prediction-response composite descriptor includes: Input the unlabeled samples into each model to obtain the probability vector and predicted class identifier of the model; For each model, the gradient of the output score corresponding to the predicted category identifier with respect to the feature map of the last convolutional layer is calculated, and the feature map is summed with channel weights using the gradient to generate the category-sensitive spatial response map of the model. The category-sensitive spatial response map is binarized to obtain a response activation region mask; The probability vector of each model, the predicted category identifier, and the response activation region mask are combined to form the prediction-response composite descriptor of that model for that sample.

5. The automatic data annotation method based on multi-model consistency verification according to claim 4, characterized in that, The step of identifying samples whose prediction consistency score exceeds a preset first threshold and whose response region overlap ratio simultaneously exceeds a preset second threshold as suspected collusion samples includes: The prediction consistency score is obtained by dividing the number of models with the same predicted category identifier among all K models by K. For the response activation region masks of all models, calculate the cross-union ratio of pairs, and take the average cross-union ratio of all model pairs as the response region overlap ratio; When the prediction consistency score is greater than the first threshold and the response region overlap ratio is greater than the second threshold, the sample is determined to be the suspected collusion sample. When the prediction consistency score is greater than the first threshold but the response region overlap ratio is less than or equal to the second threshold, the multi-model prediction is determined to be independent and consistent, and the sample is marked as a high-confidence labeled sample.

6. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The step of applying a directional feature perturbation along the main direction of the shared feature subspace to each of the suspected collusion samples, and collecting the predicted change vectors of all models after the perturbation, includes: Select the model pair with the largest edge weight from the homogeneous adjacency graph of the model and extract the shared feature subspace basis matrix of the model pair; Project the input features of the suspected collusion sample onto the first principal direction of the shared feature subspace basis matrix to obtain the projection components; The projection components are scaled by a preset perturbation step size to generate a directional perturbation vector; The directional perturbation vector is superimposed on the original input of the suspected collusion sample to generate a perturbation sample; The perturbation sample is input into all K models, and the difference between the probability vectors of each model before and after the perturbation is calculated, which is denoted as the prediction change vector of each model.

7. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The calculation of the perturbation sensitivity divergence between models based on the predicted change vector includes: Calculate the L2 norm of the predicted change vector for each model, and denote it as the perturbation sensitivity of that model; Calculate the mean and standard deviation of the perturbation sensitivity for all K models; Divide the standard deviation by the mean to obtain the coefficient of variation of the perturbation sensitivity, which is denoted as the perturbation sensitivity divergence. When the perturbation sensitivity divergence is lower than the divergence threshold, it is determined that each model responds synchronously to the perturbation in the shared bias direction, thus confirming the collusion effect.

8. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, For the non-collusion suspect samples, a weighted voting process is performed using the decision independence weight to generate labeled tags, including: The predicted category labels of each model for the non-collusion suspect sample are summarized, and the set of supporting models for each candidate category is counted. For each candidate category, the decision independence weights of each model in its supporting model set are summed to obtain the weighted vote value for that candidate category. The candidate category with the largest weighted vote value is selected as the label for the sample; The ratio of the largest weighted vote value to the sum of the weighted vote values ​​of all candidate categories is used as the confidence level of the label.

9. The automatic data annotation method based on multi-model consistency verification according to claim 1, characterized in that, The step of performing a global annotation consistency check on samples with the same annotation label and outputting the final annotation dataset includes: All labeled samples are grouped according to their labels. The mean vector of the intermediate layer feature representation of each model is extracted for each sample in each group, and the feature dispersion within the group is calculated. When the dispersion of the features within a group exceeds the dispersion threshold, hierarchical clustering is performed on the samples within the group, and samples whose feature distance from the majority of samples within the group exceeds the abnormal distance threshold are marked as potential mislabeled samples. The confidence level of the potentially mislabeled samples was downgraded to the level requiring further review. All annotation labels, confidence levels, and verification marks are integrated to output the final annotated dataset.

10. The automatic data annotation method based on multi-model consistency verification according to claim 6, characterized in that, The step of selecting the model pair with the largest edge weight from the homogeneous adjacency graph of the models and extracting the shared feature subspace basis matrix of the model pair includes: Traverse all edges in the homogeneous adjacency graph of the model and select the two model nodes connected by the edge with the largest edge weight as the model pair with the largest edge weight; Extract the feature subspace basis matrices of each of the two models in the model pair respectively; Perform joint singular value decomposition on the two eigenspace basis matrices to extract the top p joint singular vectors corresponding to the maximum singular value; The matrix formed by the first p joint singular vectors is used as the basis matrix of the shared feature subspace.

Images