Advanced active learning system based on multi-strategy fusion and sampling method

Abstract

The invention relates to the technical field of machine learning, and discloses an advanced active learning system and sampling method based on multi-strategy fusion, and the system comprises a feature module which is used for extracting the feature representation of an unlabeled sample; the strategy module comprises a plurality of learnable strategy agents and is used for evaluating sample information amount in parallel from different targets such as classification uncertainty, spatial diversity and model influence degree; the discrimination module dynamically fuses each strategy score and outputs sample selection probability distribution through a coordination network based on an attention mechanism; and the optimization module is used for generating element gradient signals according to the performance improvement of the main task model on the verification set, and jointly optimizing all the modules through the element gradient signals, so that multi-strategy dynamic intelligent collaboration and system endogenous evolution are realized, and the sample selection efficiency and the model performance can be improved.

Description

Technical Field

[0001] This application relates to the technical field of machine learning, and discloses an advanced active learning system and sampling method based on multi-strategy fusion. Background Technology

[0002] Active learning intelligently selects the most valuable samples for labeling to minimize labeling costs and maximize model performance. However, existing advanced active learning techniques have several shortcomings in achieving multi-strategy fusion. For example, the multi-strategy fusion mechanism is rigid and lacks adaptability: existing techniques typically use pre-set fixed weights or simple rules to combine uncertain and diverse strategies. Static fusion methods cannot perceive or adapt to the dynamic changes in the learning state of the main task model during training or sudden changes in the distribution of external data. When the model needs the most diversity in the exploration phase, giving too high a weight to uncertainty, or vice versa, both lead to low sample selection efficiency. Existing strategies evaluation usually relies on intermediate results from the model output or independently calculated data statistics, without establishing a direct and optimizable connection. This means that the selected samples may be useful, but their contribution to the actual improvement of model performance is limited. Although some existing technologies attempt to fuse uncertain and diverse strategies, they all use fixed weights or simple rule fusion methods and do not consider the dynamic changes in the state of the main task model during training. At the same time, existing strategies evaluation networks and feature extraction networks lack a joint optimization mechanism. Feature representation is only optimized for the main task and cannot adapt to the needs of sample information evaluation. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of the embodiments of this application and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents, and such simplifications or omissions should not be construed as limiting the scope of this application.

[0004] The advanced active learning system and sampling method based on multi-strategy fusion provided in this application aim to solve the technical problems of rigid fusion mechanisms, low computational efficiency, poor scenario adaptability, and fragmented optimization objectives in existing technologies. It realizes dynamic intelligent collaboration of multiple strategies and endogenous evolution of the system. At the same time, it adapts to large-scale unlabeled sample pools through lightweight design and supports classification and regression tasks and class imbalanced dynamic sample pools through multi-scenario adaptation modules. Ultimately, it improves sample selection efficiency, reduces labeling costs, and improves the generalization performance of the main task model.

[0005] On the one hand, this application provides an advanced active learning system based on multi-policy fusion, including a feature module, a policy module, a discrimination module, and an optimization module; The feature module is used to extract features from the unlabeled sample pool and output sample feature representations; The strategy module includes multiple different strategy agents. Each strategy agent receives the sample features, evaluates the information content of the sample from different underlying objectives, and outputs a strategy score vector. The different underlying objectives include evaluating the classification uncertainty of the sample, the diversity of the feature space, and the influence of the main task model. The discrimination module receives the policy score vectors output by all policy agents and the current main task model state. It evaluates and merges the policy scores through a coordination network, predicts the performance improvement of the main task model by different sample selection schemes, and outputs the sample selection probability distribution. The optimization module generates a meta-gradient signal based on the improvement of the main task model on the validation set after the selected samples are labeled.

[0006] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The unlabeled sample pool is a collection of machine learning samples that have not yet received labeling information; The feature module performs feature extraction through a feature encoding network trained independently of the main task model. The feature encoding network is pre-trained using self-supervised learning or an auxiliary task related to the main task, mapping the original samples into fixed-dimensional feature representations.

[0007] As a preferred embodiment of the advanced active learning system based on multi-strategy fusion proposed in this application, wherein: Each policy agent in the policy module is a lightweight, learnable neural network. Each policy agent independently receives the same batch of sample feature representations output by the feature module, and processes them through its own independent input layer and feature processing layer to adapt to a specific evaluation target.

[0008] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The process by which each policy agent evaluates the amount of information in the samples from its underlying objective and outputs a policy score vector includes parallel computation, independent mapping, and vector generation. In the parallel computing, each policy agent calculates the input sample feature set in parallel according to the target and generates an initial evaluation value. In the independent mapping, each policy agent maps the initial evaluation value to a standardized, comparable scoring range through a fully connected network layer; The generated vector is applied to the entire unlabeled sample pool. Each policy agent outputs a policy score vector, where each dimension of the policy score vector corresponds to the standardized score of a sample, which is used to quantify the amount of information about the current sample from the perspective of the agent's target.

[0009] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The strategy module includes a classification uncertainty agent, a feature space diversity agent, and a main task model influence degree agent. The classification uncertainty agent works by analyzing the predicted probability distribution obtained after the sample features are processed by the main task model classifier; by calculating the entropy of the distribution and normalizing the result, it is used as its policy score. The higher the score, the more uncertain the sample classification is. The feature space diversity agent works by analyzing the distribution density of sample features in the feature space; The average distance from the feature of the sample to the feature center of the labeled sample set or the selected sample set in this batch is calculated, and the distance value is normalized as its strategy score. The higher the score, the sparser the feature region where the sample is located and the greater the contribution of diversity. The main task model influence agent works by estimating the label and learning the expected impact of the sample on the parameters of the main task model. It adopts a lightweight approximation method for training loss gradients, only calculating the loss gradient vector of the last layer of the main task model classifier. It randomly selects a subset of the large unlabeled pool to calculate the gradient and generalizes it to the full sample pool. It also estimates the expected influence norm of the loss gradient vector on the overall loss gradient of the main task model on the validation set. After normalizing this norm, it is used as its policy score. The higher the score, the greater the expected impact of the sample on the model update.

[0010] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The coordination network in the discrimination module is a learnable neural network with an attention mechanism; All strategy score vectors output by the strategy module are concatenated with the context vector reflecting the current main task model state to form a comprehensive input vector; The coordination network processes the integrated input vector through an attention subnetwork, calculates and outputs the attention weight corresponding to each policy agent; the magnitude of the attention weight represents the relative importance of the evaluation information provided by the corresponding policy agent in the current main task model state. Using the attention weights as coefficients, the strategy scoring vector is weighted and summed to obtain the fused global scoring vector; The global score vector is mapped to a normalized selection probability for each sample in the unlabeled sample pool through a decision output layer, thus forming the sample selection probability distribution. The parameters of the coordination network, together with the parameters of the feature module and the strategy module, are jointly optimized end-to-end using meta-gradient signals from the optimization module.

[0011] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The meta-gradient signal is backpropagated through a differentiable path and used to jointly update the parameters of the discrimination module, all policy agent modules, and the feature module.

[0012] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The optimization module generates the meta-gradient signal using the following methods: The selected samples are labeled and added to the training set, and the main task model is updated to obtain a new model; the loss function values ​​of the main task model before the update and the new model on a fixed validation set are calculated respectively, and the difference is used as the performance improvement of the main task model; The actual improvement is constructed as a meta-gradient signal representing the merits of the sample selection probability distribution output by the discrimination module using a differentiable approximation method; the meta-gradient signal is directly related to the contribution of the learnable parameters in the discrimination module, policy module, and feature module to the final performance improvement; The meta-gradient signal is distributed to the coordination network of the discrimination module, each policy agent in the policy module, and the feature module through a backpropagation path; A gradient descent step is performed using the meta-gradient signal to generate the optimal sample selection decision using the jointly optimized parameters of the discrimination module, policy module, and feature module.

[0013] As a preferred embodiment of the advanced active learning system based on multi-policy fusion proposed in this application, wherein: The method for generating the meta-gradient signal by the optimization module includes: An approximate estimate of the impact of single-sample annotation on the performance of the main task model on the validation set is constructed, and the directional consistency between the overall loss gradient of the validation set and the loss gradient of a single sample is used as a measure of the expected contribution of the sample; the higher the directional consistency, the greater the expected improvement in the generalization performance of the model after annotating the sample. Using the sample selection probability distribution output by the discrimination module as weights, the expected contributions of all unlabeled samples are weighted and summed to construct a differentiable substitution loss function. Batch sampling without replacement is decomposed into multiple sequential single-sample sampling steps. In each step, a selection result is generated based on the currently unselected sample and its current selection probability distribution. The selection result retains the characteristic of being differentiable. In the forward propagation phase, discrete sampling results are used to participate in the main task model update, and in the back propagation phase, continuous gradient information is used to update parameters, so that the meta-gradient signal can be completely transmitted back through the sampling operation. The derivative of the substitution loss function with respect to the sample selection probability distribution is used to generate a meta-gradient signal; each component of the meta-gradient signal is used to indicate the adjustment of the selection probability of the corresponding sample: the selection probability of samples with large expected contributions should be increased, and the selection probability of samples with small expected contributions should be decreased. The meta-gradient signal is transmitted sequentially through the backpropagation path to the coordination network of the discrimination module, the policy agents of the policy module, and the feature encoding network of the feature module, driving the joint update of the parameters of each module.

[0014] This application provides an advanced active learning sampling method based on multi-policy fusion, including: S1. Extract features from the original samples in the unlabeled sample pool using a feature encoding network to obtain sample feature representations; S2. Input the sample feature representation into multiple different policy agents; each policy agent evaluates the information content of the sample in parallel based on different underlying objectives, and generates its own policy score vector; the underlying objectives include evaluating the classification uncertainty, feature space diversity and main task model influence of the sample; S3. Obtain the policy score vectors output by all policy agents and the current main task model state; dynamically evaluate and fuse the policy score vectors through a coordination network to predict the improvement of the main task model performance by different sample selection schemes, and output the sample selection probability distribution; select a batch of samples to be labeled from the unlabeled sample pool using a random sampling method without replacement based on the distribution, and renormalize the probability distribution of the remaining samples after sampling. S4. Label the selected sample batch and use it to update the main task model; generate a meta-gradient signal based on the performance improvement of the updated main task model on the validation set; backpropagate the meta-gradient signal through a differentiable path to jointly optimize the learnable parameters of the coordination network, all policy agents, and the feature encoding network.

[0015] The beneficial effects of this application are as follows: This application enables the system to calculate and adjust the attention weights of each policy agent based on the context vector of the current main task model by setting a discrimination module, thereby achieving real-time adaptation of the fusion strategy. This allows the sample selection process to respond sensitively to changes in the model learning stage and the data environment, and the selection logic no longer follows traditional presets, thus improving the accuracy of the selection.

[0016] This application converts the actual performance improvement of the main task model on the validation set into a global meta-gradient signal in a differentiable manner by setting an optimization module. At the same time, it backpropagates and jointly optimizes the parameters of the feature module, all policy agents and the discriminator module, forcing the entire system to co-evolve to improve the performance of the main task model: the feature representation is optimized to be more conducive to distinguishing information, thus solving the problem of the disconnect between the evaluation criteria and the final effect.

[0017] The higher the quality of the selected samples in this application, the faster the model improves. The improvement in model performance, in turn, guides the system to optimize the selection strategy more effectively through meta-gradients. Under a limited labeling budget, it can continuously select the samples that contribute the most to the improvement of the model's generalization ability. When the same model accuracy is achieved, the number of labels required is reduced, thereby improving data utilization efficiency and the final performance of the model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained through these drawings without creative effort. Wherein: Figure 1 A schematic diagram of the structure of an advanced active learning system based on multi-strategy fusion provided in this application; Figure 2 A flowchart of an advanced active learning sampling method based on multi-strategy fusion provided for this application; Figure 3 A schematic diagram illustrating the collaborative operation of the main task model of an advanced active learning system based on multi-strategy fusion, provided for this application; Figure 4 A data flow diagram of an advanced active learning system based on multi-policy fusion provided in this application; Figure 5 The image shows the detection data results of an advanced active learning system based on multi-strategy fusion, which is provided in this application. Detailed Implementation

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this application. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0022] Example 1 like Figure 1 and Figure 4 As shown, an advanced active learning system based on multi-policy fusion aims to maximize system resource utilization and minimize task latency. It includes: a feature module, a policy module, a discrimination module, and an optimization module; wherein: The feature module is used to extract features from the unlabeled sample pool and output sample feature representations; The unlabeled sample pool is a collection of machine learning samples that have not yet received labeling information; The feature module performs feature extraction through a feature encoding network trained independently of the main task model. The feature encoding network is pre-trained using self-supervised learning or an auxiliary task related to the main task, mapping the original samples into fixed-dimensional feature representations.

[0023] To avoid mismatch between feature representation and evaluation target caused by the main task model update, after every few rounds of active learning, the feature encoding network is actively aligned with the feature extraction layer of the main task model. The alignment method is to use labeled samples as a bridge to keep the output of the feature encoding network similar to the output of the feature extraction layer of the main task model.

[0024] In each round of active learning, the similarity between the feature encoding network and the feature extraction layer output of the main task model is calculated. When the similarity continues to decrease and exceeds the safety value, it is determined that representation drift has occurred and alignment update is triggered immediately, rather than waiting for a fixed period.

[0025] The initial pre-training of the feature encoding network employs an auxiliary task related to the main task, enabling it to learn feature representations with strong generalization ability without relying on labeled data, thus reducing the need for frequent alignment.

[0026] Specifically, the feature module includes a feature encoding network, which is independent of the main task model. The feature encoding network employs a learnable neural network structure. For image classification and regression tasks, a multi-layer convolutional neural network is used, consisting of 3 convolutional layers with a 3×3 kernel size and a stride of 1, 1 and 2 pooling layers (max pooling), and 2 fully connected layers with a 2×2 kernel size, stacked together to output a 512-dimensional fixed feature representation. For text classification and regression tasks, a lightweight Transformer is used with a 6-layer encoder, 8-head self-attention, and 256 hidden layers, ultimately outputting a 256-dimensional fixed feature representation. For regression tasks, the input of the feature encoding network... The outgoing layer activation function uses linear activation, while Softmax activation is used for classification tasks. The feature encoding network is pre-trained through self-supervised learning: image samples are used for rotation angle prediction tasks with rotation angles of 0°, 90°, 180°, and 270°, batch size of 64, and training epochs of 100; text samples are used for masked language model tasks with a mask ratio of 15%, batch size of 32, and training epochs of 80. After pre-training, the initial values ​​of the feature encoding network parameters are fixed. During the active learning iteration process, the parameters are updated every 3 iterations. When the accuracy improvement of the main task model on the validation set is less than 0.5% for two consecutive iterations, the parameter update of the feature encoding network is stopped.

[0027] The computational cost of forward propagation in the feature encoding network is controlled to within 8% of the computational cost of a single inference in the main task model. Lightweighting is achieved through convolution kernel decomposition of 1×1+3×3 and sparse attention mechanism of Transformer layer, which is suitable for the computational needs of large-scale sample pools.

[0028] For imbalanced datasets, feature encoding networks can preferentially introduce class weight factors during the pre-training stage. For dynamic sample pools, feature encoding networks can preferentially adopt an incremental pre-training mechanism, performing an incremental pre-training when the proportion of newly added samples exceeds 10% of the total sample pool.

[0029] Each sample in the unlabeled sample pool is independently input into the feature encoding network. After forward propagation, the corresponding feature representation is output. This feature representation serves as the unified input for the information content evaluation of the current sample in the policy module.

[0030] The parameters of the feature encoding network are initialized through pre-training, which is decoupled from the main task model. The pre-training task can be self-supervised learning or an auxiliary task related to the main task. During the active learning iteration, the feature encoding network receives the meta-gradient signal generated by the optimization module and updates its network parameters together through the backpropagation algorithm, so that the extracted feature representation gradually adapts to the goals of multi-strategy evaluation and sample selection.

[0031] The feature encoding network is connected to the policy module through a tensor data interface. The output feature representation data structure is a matrix of shape (number of samples, feature dimension), which is used to support parallel processing of each agent in the policy module.

[0032] In this application, a preferred implementation method of a feature coding network includes: Specific implementation methods of feature coding networks The feature encoding network uses a multi-layer convolutional neural network as its basic architecture. This network consists of several convolutional layers, pooling layers, and fully connected layers stacked sequentially. The convolutional layers extract local features from the input samples, the pooling layers progressively reduce the spatial size of the feature maps and enhance feature translation invariance, and the fully connected layers map the multi-dimensional feature maps output by the convolutional parts into fixed-length feature vectors. For image samples, the network input is the original pixel matrix. After successive convolutions and pooling, the final output is a feature vector with dimensions of 256 or 512. 512 dimensions are preferred for image samples, and 256 dimensions are preferred for text samples. For text samples, a Transformer structure can be used, extracting contextual semantic features through multi-head self-attention layers and positional encoding, also outputting a fixed-dimensional feature vector.

[0033] This feature encoding network is pre-trained independently of the main task model. The pre-training phase employs self-supervised learning, such as applying tasks like rotation prediction and contrastive learning to image samples, enabling the network to learn general feature representations of the samples without requiring labeled data. After pre-training, the parameters of the feature encoding network are used as initial values ​​for subsequent active learning processes.

[0034] In each iteration of active learning, the feature encoding network receives all samples from the unlabeled sample pool, computes the feature representation of each sample through forward propagation, and outputs it to the policy module in matrix form. This feature representation is decoupled from the training state of the main task model, providing a unified input source for each policy agent.

[0035] The feature encoding network is continuously updated during active learning. After the optimization module generates the meta-gradient signal, it propagates the gradient back layer by layer to the parameters of each layer of the feature encoding network through the backpropagation algorithm. Specifically, the meta-gradient signal first acts on the coordination network of the discriminator module, then propagates back along the input path of the coordination network to each agent in the policy module, and finally propagates to the output of the feature encoding network through the input interface of the policy module. The feature encoding network updates its parameters based on this gradient signal, making the subsequently extracted features more conducive to the policy module's accurate evaluation of sample information. The entire update process is decoupled from the update of the main task model, and the feature encoding network, policy module, and discriminator module form an end-to-end joint optimization system.

[0036] The strategy module includes multiple different strategy agents. Each strategy agent receives the sample features, evaluates the information content of the sample from different underlying objectives, and outputs a strategy score vector. The different underlying objectives include evaluating the classification uncertainty of the sample, the diversity of the feature space, and the influence of the main task model. Each policy agent in the policy module is a lightweight, learnable neural network. Specifically, the lightweight learnable neural network is a neural network structure with reduced parameter count and computational complexity compared to the main task model.

[0037] For example, the absolute number of network parameters for each policy agent does not exceed 100,000, of which the number of parameters for the classification uncertainty agent is preferably 50,000, the number of parameters for the feature space diversity agent is preferably 30,000, and the number of parameters for the main task model influence agent is preferably 100,000; the forward propagation computation does not exceed 10% of the computation of a single inference of the main task model, and the lightweight neural network is set up to enable the policy module to run multiple evaluation policies in parallel without increasing the computational burden of the system.

[0038] Furthermore, each policy agent adopts a multilayer perceptron structure, consisting of an input layer, several hidden layers, and an output layer. The number of neurons in the input layer is consistent with the feature dimension output by the feature module, and it is used to receive the feature representation of the sample. The number of hidden layers is set to 1 to 3, and the number of neurons in each layer decreases layer by layer, for example, from 128 to 64 and then to 32. The activation function is ReLU or GELU to introduce non-linear mapping capability. The number of neurons in the output layer is 1, and it is used to output the score value of a single sample.

[0039] Furthermore, each policy agent, feature module, and discriminator module jointly receive the meta-gradient signal generated by the optimization module for joint training, without setting an independent loss function or pre-training process. In each round of active learning iteration, the meta-gradient signal is backpropagated to the output layer of each policy agent through the discriminator module, thereby updating its network parameters layer by layer and gradually learning the scoring mapping relationship for a specific evaluation target.

[0040] Each policy agent independently receives the same batch of sample feature representations output by the feature module, and processes them through its own independent input layer and feature processing layer to adapt to a specific evaluation target.

[0041] To prevent the functions of the policy agents from converging during the joint optimization process, a preferred example strategy can be adopted in this application: during joint optimization, an orthogonality constraint is imposed on the output mapping of each policy agent, so that the score vectors output by different agents remain statistically independent, forcing each agent to learn different information content evaluation criteria.

[0042] After each round of active learning, the statistical correlation between the output scores of each policy agent is calculated. When the scoring behaviors of any two agents are highly correlated, the constraint strength on their differences is automatically increased, prompting them to restore their respective evaluation priorities.

[0043] Each policy agent employs an independent learning rate adjustment mechanism, dynamically adjusting the update magnitude based on the stability state of its own gradient update, so that different agents maintain their own learning rhythm during training.

[0044] Each policy agent has an independent input layer. The structure of the independent input layer is exactly the same as the input layer of other policy agents, that is, the number of neurons is equal to the feature dimension output by the feature module. However, in terms of parameters, each input layer has an independent weight matrix and bias vector and does not share parameters with each other. Even if two agents receive the same sample feature representation, they will get different intermediate feature vectors after linear transformation through their respective input layers.

[0045] After the input layer of each policy agent, an independent feature processing layer is set. The feature processing layer consists of one or more fully connected layers. The feature processing layer of each agent can be configured independently according to the complexity of the evaluation target. For example, the classification uncertainty agent can be set with fewer layers, such as 1 layer, while the main task model influence agent can be set with more layers, such as 2 layers, because the evaluation logic is more complex. The parameters of each feature processing layer are completely independent.

[0046] In this application, the parameters of each agent are optimized only for its specific evaluation objective, avoiding parameter interference between different evaluation objectives. For example, the weights learned by an agent with classification uncertainty will not affect the scoring logic of a diverse agent; each agent can flexibly set the network depth and width according to the complexity of its own evaluation objective, realizing on-demand allocation of computing resources and avoiding redundancy or insufficiency caused by a uniform structure; since there is no parameter dependency between agents, forward propagation can be executed in parallel during the inference phase, making full use of the parallel capabilities of modern computing devices and improving the system response speed.

[0047] Furthermore, the specific implementation methods for each policy agent to adapt to a specific evaluation objective include: The strategy module includes a classification uncertainty agent, a feature space diversity agent, and a main task model influence degree agent. The goal of the classification uncertainty agent is to quantify the classification uncertainty of samples. The adaptation method is as follows: after receiving the feature representation of the sample, the agent maps it through an independent input layer and feature processing layer, and outputs a scalar value. During training, the classification uncertainty agent is optimized through the meta-gradient signal to output a score positively correlated with the classification uncertainty of the sample. Specifically, when a sample is selected and labeled and brings a high performance improvement to the main task model, the meta-gradient signal will adjust the parameters of the current agent so that it outputs a higher score for similar samples in the future; conversely, if the selected sample contributes little to the performance improvement, its score is reduced. Through multiple iterations, the classification uncertainty agent gradually learns an evaluation standard consistent with the classification uncertainty without human intervention.

[0048] The policy agent also includes a feature space diversity agent whose goal is to quantify the sparsity or representativeness of samples in the feature space. The adaptation method is as follows: the feature space diversity agent independently learns how to extract information related to diversity from sample features. During training, the meta-gradient signal guides the agent to output a score that is positively correlated with the sparsity of the sample feature space distribution. When a sample is located in a sparse region in the feature space, selecting the current class sample often brings a greater improvement in the generalization performance of the main task model. The meta-gradient signal thereby strengthens the agent's high score output for sparse region samples, enabling it to gradually learn to perceive the distribution density of samples from the feature representation.

[0049] The policy agent also includes a main task model influence agent, whose goal is to quantify the potential impact of samples on the parameters of the main task model. The adaptation method is as follows: the agent independently learns the mapping relationship between sample features and the impact of main task model parameter updates. During training, the meta-gradient signal backpropagates the actual performance improvement of the main task model to the main task model influence agent, enabling it to learn which sample features predict a large model influence. For example, when an update to the label of a sample causes a decrease in the loss of the main task model on the validation set, the meta-gradient signal enhances the agent's high-scoring response to such sample features, gradually leading to an accurate estimate of the model's influence.

[0050] The classification uncertainty agent works by analyzing the predicted probability distribution obtained after the sample features are processed by the main task model classifier; by calculating the entropy of the distribution and normalizing the result, it is used as its policy score. The higher the score, the more uncertain the sample classification is. The classification uncertainty agent receives the sample feature representation output by the feature module and the output interface of the main task model classifier.

[0051] In each iteration of active learning, all samples in the unlabeled sample pool first go through the forward propagation of the main task model. The main task model classifier outputs the predicted probability distribution of each sample. For classification tasks, the predicted probability distribution is a vector with a dimension equal to the total number of categories, and the value of each dimension is the predicted probability that the current sample belongs to the corresponding category.

[0052] Furthermore, for the i-th sample in the sample pool, its predicted probability distribution is denoted as pi=[p{i1},p{i2},...,p{iC}], where C is the total number of categories. The agent, dealing with classification uncertainty, reads the probability distribution and executes the entropy calculation formula: , Let be the entropy value of the i-th sample, and C be the total number of categories. Let be the predicted probability that the i-th sample belongs to the j-th class. The entropy value is calculated independently for each sample to obtain the entropy value of each sample. The larger the entropy value, the more uniform the probability distribution and the less uncertain the model is about classifying the sample; the smaller the entropy value, the more certain the model is about classifying the sample.

[0053] The classification uncertainty agent organizes the entropy values ​​of all samples into a tensor of shape (N, 1), denoted as the entropy vector H, where N is the total number of samples in the unlabeled sample pool. The entropy vector serves as the initial evaluation value, and its value range depends on the number of categories C: when C=10, the maximum possible entropy value is approximately 2.3026; when C=100, the maximum possible entropy value is approximately 4.6052.

[0054] The entropy vector H is input to the independent mapping network of the agent with classification uncertainty. The independent mapping network is a single-layer fully connected network containing a weight parameter wu and a bias parameter bu. The mapping network first calculates the weighted value: wu × +bu; then the result is compressed to the (0, 1) interval using the Sigmoid function, and the standardized score is output: ri = Sigmoid(wu × +bu), the standardized score is the policy score finally output by the agent with classification uncertainty. The closer the score is to 1, the higher the classification uncertainty of the sample.

[0055] Furthermore, the parameters wu and bu of the mapping network are updated through the meta-gradient signal during the active learning process. When the meta-gradient signal is backpropagated to the agent, the main network is updated at the same time. It should be noted that the main network is the entropy calculation logic, which only updates the mapping network parameters, so that the standardized score can more accurately reflect the relative size of the amount of uncertainty information in the (0,1) interval.

[0056] The feature space diversity agent works by analyzing the distribution density of sample features in the feature space; The feature space diversity agent sets a feature center vector, which is obtained by calculating the average of the feature representations of all samples in the labeled sample set. Let the labeled sample set be L, where the number of samples is M, and the feature representation of each sample be fj, then the feature center vector is... After each round of active learning and the addition of new labeled samples, the agent recalculates the feature centers, achieving dynamic updates.

[0057] When using batch selection mode, the agent maintains the feature center of the selected sample set in the current batch during the single round selection process. The feature center is initialized to empty. After each sample is selected to be added to the batch, the feature center is updated immediately to avoid the selected samples being too concentrated in a single round.

[0058] For the i-th sample in the unlabeled sample pool, its features are represented as: The agent calculates the Euclidean distance from fi to the feature center c: That is, calculate the sum of the squares of the differences in each dimension of the two feature vectors, and then take the square root. The calculation is performed independently for each sample to obtain the distance value vector d, which has the shape (N, 1). The larger the distance value, the farther the feature representation of the sample is from the feature center of the existing labeled sample set, and the sparser its region is.

[0059] The distance values ​​of all samples are organized into a distance vector D.

[0060] The distance vector D is input to an independent mapping network of the feature space-diversity agent. This mapping network is also a single-layer fully connected network, containing weight parameters wd and bias parameters bd. The calculation is performed as follows: ri = Sigmoid(wd × ... +bd), outputs a standardized score. The closer the score is to 1, the farther the sample feature is from the feature center of the labeled sample set, and the greater the contribution to diversity.

[0061] The mapping network parameters wd and bd are updated through the meta-gradient signal, which establishes a correlation between the standardized score and the actual contribution of the sample to the improvement of the model's generalization performance. When selecting samples that are far away but have little actual contribution, the meta-gradient signal will adjust the mapping parameters to reduce the score weight of such samples.

[0062] The average distance from the feature of the sample to the feature center of the labeled sample set or the selected sample set in this batch is calculated, and the distance value is normalized as its strategy score. The higher the score, the sparser the feature region where the sample is located and the greater the contribution of diversity. The main task model influence agent works by estimating the label and learning the expected impact of the sample on the parameters of the main task model. It adopts a lightweight approximation method for training loss gradients, only calculating the loss gradient vector of the last layer of the main task model classifier. It randomly selects a subset of the large unlabeled pool to calculate the gradient and generalizes it to the full sample pool. It also estimates the expected influence norm of the loss gradient vector on the overall loss gradient of the main task model on the validation set. After normalizing this norm, it is used as its policy score. The higher the score, the greater the expected impact of the sample on the model update.

[0063] It should be noted that the size of the subset used for gradient calculation is dynamically determined based on the total size of the unlabeled sample pool. When the sample pool is large, representative samples are drawn according to a preset fixed ratio. When the sample pool is small, the sampling ratio is increased. The sampling process adopts a stratified sampling method to make the distribution of each category of samples in the subset basically consistent with the full sample pool, thus avoiding gradient estimation bias caused by class imbalance.

[0064] After obtaining the gradient vectors of the subset samples, the gradient information is extended to the full sample pool. Specifically, the gradient vector of each sample in the subset is used as a representative point. The similarity between the features of each sample in the full sample pool and these representative points is calculated. The higher the similarity, the closer the estimated gradient of the sample is to the gradient of the representative point. The approximate gradient distribution of the full sample pool is obtained through this weighted method.

[0065] A preferred example: This application may also introduce an error compensation mechanism, which records the true gradient of the full pool randomly sampled in some rounds, compares it with the subset estimation result to calculate a bias factor, and updates the bias factor by moving average to correct the subset estimation result of the round.

[0066] For the i-th sample in the unlabeled sample pool, its features are represented as: The main task model influence agent first obtains the gradient of the loss function of the main task model under the current parameters θ. Specifically, this is done by processing the sample features... Input the main task model classifier to obtain the predicted probability distribution; calculate the cross-entropy loss for this sample. Regarding the loss The gradient vector is obtained by calculating the gradient of the parameters of the last layer of the classifier in the main task model. The gradient vector reflects the direction and magnitude of the direct impact of the sample on the model parameter update.

[0067] The agent simultaneously calculates the overall loss gradient of the main task model on the validation set, let the validation set be V, containing V samples. The agent calculates the loss for each sample in the validation set sequentially, and sums them to obtain the total loss. ;right The gradient vector is obtained by calculating the gradient of the parameters of the last layer of the classifier in the main task model. This gradient vector reflects the overall direction of the model's optimization of the validation set performance under the current state.

[0068] Specifically, the methods for calculating the sample loss gradient vector include:

[0069] in, Let be the loss gradient vector of the i-th sample with respect to the classifier parameters; The parameters of the last layer of the classifier for the main task model; The loss function; Let i be the feature vector of the i-th sample; This is the model's predicted output for the i-th sample; This is the symbol for gradient calculation.

[0070] The method for calculating the overall loss gradient vector includes:

[0071] in, To validate the gradient vector of the overall loss of the validation set with respect to the classifier parameters; This represents the total loss of the model on the validation set with the current parameters θ.

[0072] The agent calculates the sample gradient vector Gradient vector of validation set The cosine similarity between the samples is calculated, and this similarity is multiplied by the magnitude of the sample gradient vector to obtain the expected influence norm. The estimated value reflects the degree of consistency between the sample gradient and the optimization direction of the validation set. The higher the consistency and the larger the gradient magnitude, the greater the impact of the sample expectation on the model.

[0073] It is important to note that, to avoid performing full backpropagation calculations on all unlabeled samples in each iteration, the agent employs a gradient approximation strategy. Specifically, it calculates only the gradient of the last layer of the main task model's classifier, without backpropagating across all network layers. For the large unlabeled pool, a subset is randomly selected for gradient calculation, and then generalized to the entire pool using kernel density estimation. This lightweight approximation significantly reduces computational overhead while maintaining the validity of gradient direction information.

[0074] The influence norm estimates of all samples are organized into an influence vector I with shape (N, 1), which serves as the initial evaluation value. The numerical range of the influence norm depends on the scale of the gradient magnitude and may vary between different models and different training stages.

[0075] The influence vector I is input into the independent mapping network of the influence agent of the main task model, and the calculation is performed: ri=Sigmoid(wi×Ii+bi), and the standardized score is output. The closer the score is to 1, the greater the expected influence of the sample on the update of the main task model.

[0076] The mapping network parameters wi and bi are updated through meta-gradient signals. When a sample is selected and labeled, if it makes a significant contribution to improving the performance of the main task model, the meta-gradient signal strengthens the agent's high score output for such samples; if the contribution is not significant, its score is reduced, so that the agent gradually learns the ability to accurately evaluate the impact.

[0077] The process by which each policy agent evaluates the amount of sample information from its underlying objective and outputs a policy score vector includes parallel computation and independent mapping to generate vectors. In the parallel computing, each policy agent calculates the input sample feature set in parallel according to the target and generates an initial evaluation value. Specifically, the strategy module is deployed in a hardware environment that supports parallel computing, such as a computing device with a multi-core GPU or CPU cluster. The sample feature representation output by the feature module is passed to the strategy module in the form of a two-dimensional tensor. The shape of the two-dimensional tensor is (N, D), where N is the total number of samples in the unlabeled sample pool and D is the dimension of the feature representation.

[0078] Furthermore, the policy module contains K policy agents, each agent is allocated an independent computational flow or computational core on the hardware, the two-dimensional feature tensor is broadcast to all K agents simultaneously, each agent starts forward propagation computation at the same time, and each agent performs its independent neural network forward inference, including input layer transformation, hidden layer activation and output layer computation.

[0079] Furthermore, within each agent, the input layer multiplies the feature tensor with its own weight matrix and adds a bias to obtain an intermediate tensor of shape (N, H1), where H1 is the number of neurons in the first hidden layer. This intermediate tensor passes through the hidden layers sequentially, with each layer performing matrix multiplication and activation function operations. Finally, the output layer generates an initial evaluation value tensor of shape (N, 1). The numerical range of the initial evaluation value is determined by the network parameters of each agent. These network parameters can be set by those skilled in the art in conjunction with the technical content disclosed in this application, common knowledge in the field, and specific implementation scenarios. The initial default values ​​for each parameter have been specified in this application.

[0080] All agents adopt a synchronous parallel mode, that is, each agent does not wait for each other during the forward propagation process, but must complete the calculation of the initial evaluation value before sending the result to the next stage. When any agent completes the calculation, the output is temporarily stored in the video memory buffer. After all agents have completed the calculation, the initial evaluation value tensors of each agent are merged in the form of a three-dimensional tensor with shape (K, N, 1).

[0081] In the independent mapping, each policy agent maps the initial evaluation value to a standardized, comparable scoring range through a fully connected network layer; Each policy agent has an independent mapping network after its output layer. The mapping network is a single-layer fully connected network without hidden layers. The input dimension of the mapping network is 1, which is the initial evaluation value output by the agent. The output dimension is 1. The parameters of the mapping network also include a weight scalar w and a bias scalar b. Each agent maintains its own w and b independently and they are not shared.

[0082] For the i-th sample in the sample pool, the initial evaluation value of agent k is s{k,i}, and the mapping network performs the following calculations: First, multiply s{k,i} by the weight wk to obtain the weighted value; then add the weighted value to the bias bk to obtain the intermediate result; finally, compress the numerical range of the intermediate result through a fixed activation function, the Sigmoid function, whose output range is (0, 1), and the output of the mapping network is the standardized score r{k,i}=Sigmoid(wk×s{k,i}+bk).

[0083] Since all agents' mapping networks use the same Sigmoid activation function, the standardized scores output by each agent are compressed into the (0, 1) interval, eliminating the inconsistency in magnitude caused by differences in network parameters between different agents. Scores within this (0, 1) interval are comparable; the 0.7 output by the agent with classification uncertainty and the 0.7 output by the agent with diversity numerically represent the same level of information content judgment.

[0084] It is important to note that the mapping network parameters of each agent are initialized before the first run. The weight wk is initialized to 1.0 and the bias bk is initialized to 0.0. At this time, the mapping network is an identity mapping and then compressed by Sigmoid, and the output value is Sigmoid(s{k,i}). During the active learning iteration, the mapping network parameters and the agent's main network parameters receive the meta-gradient signal backpropagated by the optimization module and are updated synchronously. The meta-gradient signal is transmitted to the mapping network of each agent through the discriminant module to adjust wk and bk so that the standardized score more accurately reflects the relative size of the sample information in the interval (0,1).

[0085] The generated vector is applied to the entire unlabeled sample pool. Each policy agent outputs a policy score vector, where each dimension of the policy score vector corresponds to the standardized score of a sample, which is used to quantify the amount of information about the current sample from the perspective of the agent's target.

[0086] After each policy agent completes its independent mapping, it generates a standardized score tensor with a shape of (N, 1). The second dimension of the tensor, i.e., the score dimension, is retained to form a one-dimensional vector of length N, which serves as the policy score vector for the current agent. K agents generate a total of K policy score vectors, and the index position i of each vector corresponds to the standardized score of the i-th sample in the unlabeled sample pool.

[0087] Furthermore, during the generation of policy score vectors, all agents strictly follow the unified index order of the sample pool. The feature tensors output by the feature module are arranged according to the storage order of the samples in the unlabeled sample pool. Each agent maintains this order during parallel computation and independent mapping stages, ensuring that the policy score vectors output by different agents correspond to the same sample at the same index position.

[0088] Furthermore, the K policy score vectors are encapsulated in the form of a dictionary or a list of tensors for use by the discrimination module. A preferred encapsulation format can be: a list containing K elements, each element being a tensor of shape (N, 1); or a two-dimensional tensor of shape (K, N), where the first dimension indexes the agent and the second dimension indexes the sample.

[0089] It should be noted that each dimension of the policy scoring vector is a standardized score within the interval (0, 1). The standardized score is used to quantify the amount of information in the corresponding sample from the perspective of the agent's evaluation target: the closer the value is to 1, the greater the amount of information the agent believes in this sample; the closer the value is to 0, the smaller the amount of information. Since the scores of all agents have undergone the same standardization process, the discrimination module can directly compare and merge the scores of different agents without additional normalization operations.

[0090] This application utilizes hardware parallelism to generate initial evaluation values ​​for each agent through parallel computing. Independent mapping unifies heterogeneous scores to a comparable range through a learnable standardized network, and the generated vectors are output in a standardized data structure, ensuring independent computation of data and stable operation of each agent.

[0091] The discrimination module receives the policy score vectors output by all policy agents and the current main task model state. It evaluates and merges the policy scores through a coordination network, predicts the performance improvement of the main task model by different sample selection schemes, and outputs the sample selection probability distribution. The coordination network in the discrimination module is a learnable neural network with an attention mechanism, such as... Figure 3 As shown; The discrimination module includes a coordination network with an attention mechanism. The coordination network consists of four sub-components: an input splicing layer, an attention sub-network, a weighted fusion layer, and a decision output layer. All layers of the coordination network are learnable neural network layers. The parameters of the learnable neural network are jointly optimized end-to-end with the parameters of the feature module and the policy module by accepting the meta-gradient signal generated by the optimization module.

[0092] All strategy score vectors output by the strategy module are concatenated with the context vector reflecting the current main task model state to form a comprehensive input vector; The coordination network processes the integrated input vector through an attention subnetwork, calculates and outputs the attention weight corresponding to each policy agent; the magnitude of the attention weight is used to characterize the relative importance of the evaluation information provided by the corresponding policy agent in the current main task model state. The coordination network first receives K policy rating vectors from the policy module. Each vector has a shape of (N, 1), where N is the total number of samples in the unlabeled sample pool, and K is the number of policy agents. In this specific implementation, K is set to 3. The coordination network concatenates these K vectors along the feature dimension to form a rating matrix R of shape (N, K), where the element R{i,k} in the i-th row and k-th column represents the standardized rating of the k-th policy agent for the i-th sample.

[0093] The coordination network simultaneously acquires the current state context vector of the main task model. The context vector is constructed as follows: the loss value of the main task model in the current training epoch on the validation set, the validation set accuracy, and the total number of labeled samples since the start of active learning are collected. These three scalar values ​​are concatenated into a three-dimensional vector, denoted as... The shape is (1, 3). The three-dimensional vector reflects the learning progress and performance status of the main task model at the current moment.

[0094] Since the state context vector of the main task model is the same for the entire sample pool, the coordination network first... Copy N copies along the sample dimension to expand it into a matrix S of shape (N, 3). Then concatenate the scoring matrix R and the state matrix S along the feature dimension to form a comprehensive input vector matrix X of shape (N, K+3). Each row of the vector matrix X corresponds to a sample, which contains the score of the current sample under each strategy and the global state information of the current main task model.

[0095] Using the attention weights as coefficients, the strategy scoring vector is weighted and summed to obtain the fused global scoring vector; The attention subnetwork receives a synthesized input vector matrix X as input. The attention subnetwork consists of two parallel fully connected layers: a query generation layer and a key generation layer. The query generation layer maps the input matrix X to a query matrix Q with shape (N, dk). The key generation layer maps the input matrix X to a key matrix K with shape (N, dk), where dk is the dimension of the attention head. In a preferred example in this application, dk is set to 16.

[0096] Furthermore, the attention subnetwork calculates the dot product between the query matrix Q and the transpose of the key matrix K to obtain the attention score matrix. The expression is of shape (N, N), where the element in the i-th row and j-th column represents the attention association strength between the i-th sample and the j-th sample. To reduce computational complexity, this application adopts a sample-level aggregation method: the query vector of each sample is multiplied by its own key vector to obtain the attention score for each sample. ,in and These are the query vector and key vector for the i-th sample, respectively. This method reduces the complexity from O(N²) to O(N).

[0097] The above attention scores This reflects the importance at the sample level, but this application needs to generate attention weights at the policy level. Therefore, the attention subnetwork further sets up a third fully connected layer to integrate the input vector of each sample. Mapped to policy attention vector The shape is (K, 1), where K is the number of policy agents. The mapping is achieved through a weight matrix Wattn with a shape of (K+3, K) and a bias vector battn with a shape of (K, 1). The softmax function is used to ensure that the attention weights of each policy are normalized at the sample level, that is, for each sample i, we have .

[0098] The coordination network will coordinate the policy attention vectors of each sample. Taking the average along the sample dimension, we obtain the global attention weight vector β, which has the shape (K, 1): The global weight vector β represents the average relative importance of each policy agent to the entire sample pool under the current main task model state. The averaging operation decouples the attention weights from the number of samples, thus maintaining stability.

[0099] The coordination network uses a global attention weight β to perform a weighted summation of the score vectors of each policy, let the score vector of the k-th policy agent be . If the shape is (N, 1), then the formula for calculating the global score vector p is:

[0100] in For scalar weights, Given a vector, the weighted sum results in a vector of shape (N, 1), where each element... This represents the global score after fusion of the i-th sample.

[0101] In a preferred example, the coordination network first assigns K score vectors Stack the weights into a rating matrix R of shape (N, K). Expand the global attention weights β into a weight matrix B of shape (N, K), where each row of B is β^T. Then calculate the element-wise product: P = R⊙B, where ⊙ represents the Hadamard product. Finally, sum along the policy dimension to obtain the global rating vector g.

[0102] The global score vector is mapped to a normalized selection probability for each sample in the unlabeled sample pool through a decision output layer, thus forming the sample selection probability distribution. The decision output layer is a single-layer fully connected network. Its input is a global score vector g with shape (N, 1), and its output is a sample selection probability vector π with shape (N, 1). This layer includes a learnable temperature parameter τ to control the smoothness of the probability distribution.

[0103] The decision output layer applies a softmax function to the global score vector g and introduces a temperature parameter τ: in Let τ be the probability of the i-th sample being selected, and let τ be a learnable temperature parameter. The initial value is set to 1.0. The temperature parameter τ is optimized through the meta-gradient signal during training so that the probability distribution can be adaptively adjusted: when τ is large, the probability distribution tends to be uniform, increasing the exploratory nature; when τ is small, the probability distribution tends to be concentrated, increasing the utilization.

[0104] Coordination network output sample selection probability distribution Then, the system randomly samples from N unlabeled samples according to the distribution, selecting B samples to form a batch to be labeled. The sampling process uses sampling without replacement. After each sampling, the probability distribution is updated, the selected samples are removed and normalized again, and the probability of the remaining unselected samples is calculated. Continue until all B samples have been selected.

[0105] The parameters of the coordination network, together with the parameters of the feature module and the strategy module, are jointly optimized end-to-end using meta-gradient signals from the optimization module.

[0106] A preferred implementation example of end-to-end joint optimization: All learnable parameters of the coordination network, including the weight matrix Wattn and bias battn of the attention sub-network, the temperature parameter τ of the decision output layer, the feature encoding network parameters of the feature module, and the parameters of each policy agent in the policy module, including the main network parameters and the mapping network parameters, together constitute the learnable parameter set Θ of the system. Specifically, the backpropagation method for the meta-gradient signal generated by the optimization module is as follows: First, starting from the performance improvement of the main task model, a meta-gradient signal is constructed; the meta-gradient signal is directly applied to the decision output layer of the discriminator module to update τ; then, it continues to backpropagate to the weighted fusion layer and the attention sub-network to update Wattn and battn; next, it propagates to the policy module through the policy scoring vector to update the mapping network parameters and main network parameters of each policy agent; finally, it propagates to the feature module through the input interface of the policy module to update the parameters of the feature encoding network.

[0107] Furthermore, during backpropagation, to prevent gradient anomalies, the coordination network incorporates a gradient truncation mechanism at the attention subnetwork: when the gradient norm exceeds a preset threshold, for example, a preset threshold of 5.0, the gradient is scaled down to the current threshold. Simultaneously, layer normalization is applied after the weighted fusion layer to stabilize the training process.

[0108] In each round of active learning iteration, after the optimization module generates the meta-gradient signal, the system performs a gradient descent update on the entire parameter set Θ. The gradient descent update enables the coordination network to gradually learn to allocate reasonable policy attention weights under different main task model states, so that the fused global score vector can more accurately predict which samples contribute the most to the improvement of model performance.

[0109] The optimization module generates a meta-gradient signal based on the improvement of the main task model on the validation set after the selected samples are labeled.

[0110] The optimization module generates the meta-gradient signal using the following methods: The selected samples are labeled and added to the training set, and the main task model is updated to obtain a new model; the loss function values ​​of the main task model before the update and the new model on a fixed validation set are calculated respectively, and the difference is used as the performance improvement of the main task model; The actual improvement is constructed as a meta-gradient signal representing the merits of the sample selection probability distribution output by the discrimination module through a differentiable approximation; the meta-gradient signal is directly related to the contribution of the learnable parameters in the discrimination module, policy module, and feature module to the final performance improvement; It should be noted that the real lift is constructed as the meta-gradient signal of the sample selection probability distribution output by the discrimination module through the differentiable approximation method of first-order Taylor expansion.

[0111] The meta-gradient signal is distributed to the coordination network of the discrimination module, each policy agent in the policy module, and the feature module through a backpropagation path; A gradient descent step is performed using the meta-gradient signal to generate the optimal sample selection decision using the jointly optimized parameters of the discrimination module, policy module, and feature module.

[0112] Specifically, the feature encoding network, policy agent, and discriminator module adopt different update frequencies. The discriminator module updates at the highest frequency to quickly respond to changes in the main task model state, while the feature encoding network updates at the lowest frequency to maintain the relative stability of the feature representation.

[0113] During backpropagation, gradient changes are monitored, and when gradients are abnormal, they are actively limited to prevent excessively large parameter update steps from causing training oscillations.

[0114] Continuously monitor the performance improvement of the main task model on the validation set. When the performance improvement tends to plateau after several rounds, automatically pause the updates of the feature encoding network and policy agent, and retain only the fine-tuning capability of the discrimination module. When the main task model changes significantly, reactivate the updates of all modules.

[0115] The optimization module receives the following inputs: the selected sample batch (after labeling), the parameter state before the main task model is updated, the parameter state after the main task model is updated, the fixed validation set, and the sample selection probability distribution output by the discriminant module. The optimization module outputs a meta-gradient signal, which is distributed to the discriminant module, the policy module, and the feature module through the backpropagation path to update the learnable parameters.

[0116] Before selecting samples, the optimization module first calculates the loss value of the current main task model on a fixed validation set. Let the current parameters of the main task model be... The validation set is fixed at V, containing V samples. The optimization module inputs all samples from the validation set into the main task model sequentially, calculates the loss for each sample, and sums them to obtain the total loss, where ℓ is the loss function, and this loss value serves as the baseline value.

[0117] The selected batches of samples, after being manually labeled, are added to the training set to form the expanded training set. The optimization module will optimize the main task model in... Training is performed on the [platform name], parameters are updated, and the training process follows a standard supervised learning workflow: [The process involves]... The model is divided into small batches and iterated multiple times until convergence or the preset number of iterations is reached. After training, the new model parameters are obtained. .

[0118] The optimization module recalculates the loss value of the main task model on the fixed validation set V. The calculation method is the same as before the update.

[0119] The optimization module calculates the difference in validation set loss as the performance improvement:

[0120] A positive ΔL indicates improved model performance (reduced loss), while a negative ΔL indicates decreased performance. This difference directly reflects the actual contribution of the selected batch of samples to the generalization ability of the model in the main task.

[0121] The performance improvement ΔL cannot be directly used to generate the primitive gradient signal because the sample selection process involves discrete sampling operations, which are not differentiable. The optimization module needs to use a differentiable approximation method to transform ΔL into a propagable gradient signal.

[0122] Furthermore, the optimization module constructs an alternative loss function. This is used to approximate the dependence of ΔL on the sample selection probability distribution. Specifically, it is implemented by treating the sample selection probability distribution π as a decision variable and defining the substitution loss:

[0123] in Let be the probability that the i-th sample is selected. Let be the performance improvement of the main task model on the validation set after selecting and labeling the i-th sample alone. It cannot be calculated directly because it is impossible to train a model separately for each unlabeled sample.

[0124] Furthermore, the optimization module uses a first-order Taylor expansion to approximate the effect of a single sample. Specifically, after adding the i-th sample to the training set, the first-order approximation of the change in model parameters is: Where η is the learning rate, Let be the loss gradient for sample i. Then the change in the validation set loss can be approximated as:

[0125] This approximation requires no actual model training, has low computational cost, and is completely differentiable.

[0126] Substituting the above approximation into the substitution loss function, an example of constructing a preferred substitution loss function includes: The optimization module first obtains the parameters of the current main task model. Then, all samples in the fixed validation set V are sequentially input into the main task model, the loss for each sample is calculated, and the total loss is obtained by summing them. The optimization module takes the derivative of the total loss with respect to the model parameters θ to obtain the gradient vector. The gradient vector reflects the sensitivity of the current model parameters to the performance on the validation set.

[0127] For the i-th sample in the unlabeled sample pool, the optimization module obtains the feature representation of that sample. and its pseudo-labels The sample is generated based on predictions from the main task model. Then, the loss for that sample is calculated. The gradient vector with respect to the model parameters θ The gradient vector reflects the direction and magnitude of the influence of the sample on the model parameter update.

[0128] The optimization module calculates the dot product of the gradient vector of the validation set and the gradient vector of the samples. :

[0129] A larger dot product value, i.e. a larger positive value, indicates that the gradient direction of the sample is more consistent with the optimization direction of the validation set, and the greater the potential to improve the generalization ability of the model after labeling the sample; a negative dot product value indicates that the sample may impair the performance of the validation set.

[0130] According to the first-order Taylor expansion, after adding the i-th sample to the training set and training for one step, the approximate change in the validation set loss is:

[0131] The negative sign indicates the direction of gradient descent; updating parameters in the direction of negative gradient will reduce the loss.

[0132] The optimization module will optimize the sample selection probability distribution. As a decision variable, construct an alternative loss function:

[0133] Substitution The result was:

[0134] The physical meaning of the substitution loss function is the probability of sample selection. When focusing on samples with larger dot product values, Larger; when focusing on samples with smaller dot product values, Smaller. Therefore, maximize. This is equivalent to selecting the sample that contributes the most to improving the performance of the validation set.

[0135] The optimization module optimizes the alternative loss function. Regarding the probability of sample selection Taking the derivative, we obtain the elementary gradient signal: That is, the meta-gradient signal is a vector of length N, and its i-th component is equal to η⋅doti. The meta-gradient signal directly indicates the adjustment of the selection probability of each sample: the selection probability of samples with large dot product values ​​should be increased, and the selection probability of samples with small dot product values ​​should be decreased.

[0136] Optimization module The gradient of the sample selection probability distribution π output by the discrimination module is calculated to obtain the meta-gradient signal:

[0137] That is, the meta-gradient signal is a vector of length N, and each component corresponds to the gradient contribution value of a sample.

[0138] If there are B samples in a batch, the optimization module approximates the impact of batch selection as the sum of the gradient contributions of the selected samples and uses Monte Carlo estimation to smooth the sampling process. In the specific implementation, REINFORCE or Gumbel-Softmax techniques are used to continuously relax the discrete samples, so that the meta-gradient signal can propagate back through the sampling operation.

[0139] The specific method for establishing the backpropagation path is as follows: Meta gradient signal First, it applies to the decision output layer of the discrimination module. This layer outputs the sample selection probability distribution π, whose parameter is the learnable temperature parameter τ. The optimization module then calculates... Through the chain rule:

[0140] in The derivative of temperature is calculated using softmax.

[0141] The meta-gradient signal is backpropagated to the attention sub-network. The global attention weight β is output by the attention sub-network, and v is determined by β and the policy score. The optimization module calculates... Then, the forward propagation formula of the attention subnetwork is used to calculate... and .

[0142] The meta-gradient signal propagates to the policy module through the policy score vector. Let the score vector of the k-th policy agent be... Its contribution to the elementary gradient is Where g is the global score vector, the optimization module will Backpropagation is performed to the mapping network and the main network of each policy agent to update parameters wk, bk, and the parameters of the main network.

[0143] The meta-gradient signal is ultimately propagated to the feature module through the input interface of the policy module, and the feature module outputs the feature representation. Directly affects the input of each policy agent and the calculation of the optimization module. And calculate using the chain rule ,in These are the parameters of the feature encoding network.

[0144] After the optimization module completes the calculation of the gradients of all parameters, it performs a gradient descent update on the system's learnable parameter set Θ:

[0145] Where α is the meta-learning rate, which is set to be the same as or slightly smaller than the learning rate of the main task model, and is usually between 0.001 and 0.01.

[0146] The optimization module performs a parameter update once in each round of active learning iteration. The update order is as follows: first, update the decision output layer and attention subnetwork of the discrimination module; then, update each agent of the policy module; and finally, update the feature encoding network of the feature module.

[0147] To prevent gradient explosion, the optimization module sets a gradient truncation mechanism during backpropagation: when the gradient norm of any parameter exceeds a preset threshold, the gradient is scaled to the threshold range. At the same time, the optimization module adds small-amplitude Gaussian noise with a mean of 0 and a standard deviation of 0.01 after the meta-gradient signal is generated to enhance the robustness of the optimization.

[0148] Through multiple iterations, the discrimination module gradually learns to allocate reasonable policy attention weights, and the agents in the policy module gradually converge to a scoring function that matches their evaluation target. The feature representations extracted by the feature module are also gradually optimized into a form that is more conducive to evaluating the amount of sample information. The entire system achieves end-to-end co-evolution driven by the meta-gradient signal.

[0149] like Figure 5 As shown, each row corresponds to one unlabeled sample (sample 0 to sample 4), which is the evaluation object of the active learning sample selection. Each row contains three progress bars, which correspond to the output scores of the three types of lightweight policy agents.

[0150] Sample 1 received a comprehensive score of 89.6%, the highest priority. Each dimension showed that the progress bars for uncertainty, diversity, and impact were all close to full, and all three scores were at a high level. Sample 1 simultaneously satisfies the criteria of being difficult for the model to distinguish, having unique features, and having its value enhanced after labeling. It is a high-value sample jointly recognized by the three types of policy agents, and after fusion, it obtains the highest selection probability, making it the most ideal labeled sample for active learning.

[0151] A high score in a single dimension does not equate to high priority; only samples with high scores across multiple dimensions will be prioritized by the system. Sample 0 received a comprehensive score of 81.3%, ranking as the second highest priority. The three scores across all dimensions were at a relatively high level, slightly lower than Sample 1, with no obvious weaknesses.

[0152] Sample 0 also has high uncertainty, high diversity, and high impact, making it a high-quality annotation candidate second only to Sample 1. It can be used as a supplementary annotation sample to further improve model performance.

[0153] Avoid insufficient sample coverage caused by selecting only a single optimal sample.

[0154] Sample 3 received a comprehensive score of 82.6%, placing it in the third priority category. The scores for uncertainty were moderately high across all dimensions, while the scores for diversity and impact were relatively low.

[0155] This reflects the correction effect of multi-strategy integration, avoiding the over-selection of samples with high scores in a single dimension of uncertainty, and ensuring the comprehensiveness of sample selection.

[0156] Sample 4 received a composite score of 43.3%, indicating low priority. The performance across all dimensions was characterized by uncertainty and extremely low impact scores, with only the diversity score being moderate.

[0157] This application effectively filters low-value samples, avoids invalid annotations, and reduces annotation costs.

[0158] Sample 2 received a composite score of 23.5%, the lowest priority. The results across all dimensions show extremely low scores for uncertainty, and moderately high scores for diversity and impact.

[0159] The proportion of uncertainty in active learning also verifies that multi-strategy fusion will not mistakenly select low-value samples due to high scores in a single dimension.

[0160] Example 2 like Figure 2 As shown, an advanced active learning sampling method based on multi-policy fusion includes: S1. Features are extracted from the original samples in the unlabeled sample pool through a feature encoding network to obtain sample feature representations; the feature encoding network is independently pre-trained and dynamically adjusted according to the model state and data changes, and maintains its adaptability to the main task model through a feature alignment mechanism. S2. Input the sample feature representation into multiple different policy agents; each policy agent evaluates the information content of the sample in parallel based on different underlying objectives, and generates its own policy score vector; the underlying objectives include evaluating the classification uncertainty of the sample, the diversity of the feature space, and the influence of the main task model; each agent maintains the independence of the evaluation dimensions through orthogonality constraints, and adopts an appropriate evaluation logic for different scenarios. S3. Obtain the policy score vectors output by all policy agents and the current main task model state; dynamically evaluate and fuse the policy score vectors through a coordination network, adaptively adjust the weights of each policy according to the model learning stage, predict the improvement of the main task model performance by different sample selection schemes, and output the sample selection probability distribution; select a batch of unlabeled samples from the unlabeled sample pool using a random sampling method without replacement based on this distribution, and renormalize the probability distribution of the remaining samples after sampling. S4. Label the selected sample batch and use it to update the main task model; based on the performance improvement of the updated main task model on the validation set, generate a meta-gradient signal through a differentiable approximation method; backpropagate the meta-gradient signal through a differentiable path to jointly optimize the learnable parameters of the coordination network, all policy agents, and the feature encoding network, thereby achieving closed-loop evolution of the system.

[0161] In step S2, each policy agent is a lightweight neural network, and its evaluation process includes classification uncertainty evaluation, feature space diversity evaluation, and main task model influence evaluation. Classification uncertainty assessment: Based on the predicted probability distribution of samples by the main task model, calculate its entropy value and normalize it to obtain an uncertainty score; Feature space diversity assessment: Calculate the average distance from sample features to the feature centers of the labeled sample set, and normalize the distance values ​​to obtain a diversity score; Impact assessment of the main task model: Based on the loss gradient vector of the sample, estimate the expected norm of its impact on the overall loss gradient of the main task model on the validation set, and normalize the norm to obtain the impact score.

[0162] In step S3, the coordination network is a learnable neural network based on an attention mechanism, and the coordination process includes: The score vectors of each strategy are concatenated with the state vector of the main task model and input into the attention sub-network to generate dynamic attention weights that represent the relative importance of each strategy. The global score vector is obtained by weighting and summing the policy score vectors using the attention weights. Map the global score vector to a normalized sample selection probability distribution; In step S4, the joint optimization method includes: using a differentiable approximation method to convert the performance improvement of the main task model into the meta-gradient of the parameters of the coordination network, policy agent network, and feature encoding network, and performing differentiated gradient descent updates according to the functional characteristics of each module to ensure the system's collaborative evolution.

[0163] It is important to note that the constructions and arrangements of this application shown in several different exemplary embodiments are merely illustrative. Although only two embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible without substantially departing from the novel teachings and advantages of the subject matter described in this application. These modifications may include, for example, changes in the size, dimensions, structure, shape, and proportions of various elements, as well as parameter values ​​(e.g., temperature, pressure, etc.), installation arrangements, the use of materials, colors, orientations, etc. For example, an element shown as integrally formed may be composed of multiple parts or elements, the position of elements may be inverted or otherwise altered, and the nature or number or position of discrete elements may be changed or altered. Therefore, all such modifications are intended to be included within the scope of this application. The order or sequence of any process or method steps may be changed or rearranged by alternative embodiments. Any "apparatus plus function" clause is intended to cover, and not only structurally equivalent but also equivalent structures, the structures performing the functions described herein. Other substitutions, modifications, alterations, and omissions may be made in the design, operation, and arrangement of the exemplary embodiments without departing from the scope of this application. Therefore, this application is not limited to a particular embodiment, but extends to various modifications that still fall within the scope of the appended claims.

[0164] Furthermore, in order to provide a concise description of exemplary embodiments, not all features of actual embodiments (i.e., those features that are not relevant to the best mode of performing this application as currently considered, or those features that are not relevant to implementing this application) may be omitted.

[0165] It should be understood that numerous specific implementation decisions can be made during the development of any practical implementation, such as in any engineering or design project. Such development efforts may be complex and time-consuming, but for those of ordinary skill in the art who benefit from this disclosure, the development effort will be a routine task in design, manufacturing, and production without requiring extensive experimentation.

[0166] It should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application, and all such modifications and substitutions should be covered within the scope of the claims of this application.

Claims

1. An advanced active learning system based on multi-strategy fusion, characterized in that, include: Feature module, strategy module, discrimination module, and optimization module; The feature module is used to extract features from the unlabeled sample pool and output sample feature representations; The strategy module includes multiple different strategy agents. Each strategy agent receives the sample features, evaluates the information content of the sample from different underlying objectives, and outputs a strategy score vector. The different underlying objectives include evaluating the classification uncertainty of the sample, the diversity of the feature space, and the influence of the main task model. The discrimination module receives the policy score vectors output by all policy agents and the current main task model state. It evaluates and merges the policy scores through a coordination network, predicts the performance improvement of the main task model by different sample selection schemes, and outputs the sample selection probability distribution. The optimization module generates a meta-gradient signal based on the improvement of the main task model on the validation set after the selected samples are labeled.

2. The advanced active learning system based on multi-strategy fusion as described in claim 1, characterized in that: The unlabeled sample pool is a collection of machine learning samples that have not yet received labeling information; The feature module performs feature extraction through a feature encoding network trained independently of the main task model. The feature encoding network is pre-trained using self-supervised learning or an auxiliary task related to the main task, mapping the original samples into fixed-dimensional feature representations.

3. The advanced active learning system based on multi-strategy fusion as described in claim 1, characterized in that: Each policy agent in the policy module is a lightweight, learnable neural network. Each policy agent independently receives the same batch of sample feature representations output by the feature module, processes them through its respective input layer and feature processing layer, and adapts them to the evaluation target.

4. The advanced active learning system based on multi-strategy fusion as described in claim 3, characterized in that: The process by which each policy agent evaluates the amount of sample information from the underlying objective and outputs a policy score vector includes parallel computation and independent mapping to generate vectors; In the parallel computing, each policy agent calculates the input sample feature set in parallel according to the target and generates an initial evaluation value. In the independent mapping, each policy agent maps the initial evaluation value to a standardized, comparable scoring range through a fully connected network layer; The generated vector is applied to the entire unlabeled sample pool. Each policy agent outputs a policy score vector, where each dimension of the policy score vector corresponds to the standardized score of a sample, which is used to quantify the amount of information about the current sample from the perspective of the agent's target.

5. The advanced active learning system based on multi-strategy fusion as described in claim 4, characterized in that: The strategy module includes a classification uncertainty agent, a feature space diversity agent, and a main task model influence degree agent. The classification uncertainty agent works by analyzing the predicted probability distribution obtained after the sample features are processed by the main task model classifier; by calculating the entropy of the distribution and normalizing the result, it is used as its policy score. The higher the score, the more uncertain the sample classification is. The feature space diversity agent works by analyzing the distribution density of sample features in the feature space; The average distance from the feature of the sample to the feature center of the labeled sample set or the selected sample set in this batch is calculated, and the distance value is normalized as its strategy score. The higher the score, the sparser the feature region where the sample is located and the greater the contribution of diversity. The main task model influence agent works by estimating the label and learning the expected impact of the sample on the parameters of the main task model. It adopts a lightweight approximation method for training loss gradients, only calculating the loss gradient vector of the last layer of the main task model classifier. It randomly selects a subset of the large unlabeled pool to calculate the gradient and generalizes it to the full sample pool. It also estimates the expected influence norm of the loss gradient vector on the overall loss gradient of the main task model on the validation set. After normalizing this norm, it is used as its policy score. The higher the score, the greater the expected impact of the sample on the model update.

6. The advanced active learning system based on multi-strategy fusion as described in claim 1, characterized in that: The coordination network in the discrimination module is a learnable neural network with an attention mechanism; All strategy score vectors output by the strategy module are concatenated with the context vector reflecting the current main task model state to form a comprehensive input vector; The coordination network processes the integrated input vector through an attention subnetwork, calculates and outputs the attention weight corresponding to each policy agent; the magnitude of the attention weight represents the relative importance of the evaluation information provided by the corresponding policy agent in the current main task model state. Using the attention weights as coefficients, the strategy scoring vector is weighted and summed to obtain the fused global scoring vector; The global score vector is mapped to a normalized selection probability for each sample in the unlabeled sample pool through a decision output layer, thus forming the sample selection probability distribution. The parameters of the coordination network, together with the parameters of the feature module and the strategy module, are jointly optimized end-to-end using meta-gradient signals from the optimization module.

7. The advanced active learning system based on multi-strategy fusion as described in claim 1, characterized in that: The meta-gradient signal is backpropagated through a differentiable path and used to jointly update the parameters of the discrimination module, all policy agent modules, and the feature module.

8. The advanced active learning system based on multi-policy fusion as described in claim 7, characterized in that: The optimization module generates the meta-gradient signal using the following methods: The selected samples are labeled and added to the training set, and the main task model is updated to obtain a new model; the loss function values ​​of the main task model before the update and the new model on a fixed validation set are calculated respectively, and the difference is used as the performance improvement of the main task model; The actual improvement is constructed as a meta-gradient signal representing the merits of the sample selection probability distribution output by the discrimination module through a differentiable approximation; the meta-gradient signal is directly related to the contribution of the learnable parameters in the discrimination module, policy module, and feature module to the final performance improvement; The meta-gradient signal is distributed to the coordination network of the discrimination module, each policy agent in the policy module, and the feature module through a backpropagation path; A gradient descent step is performed using the meta-gradient signal to generate the optimal sample selection decision using the jointly optimized parameters of the discrimination module, policy module, and feature module.

9. The advanced active learning system based on multi-policy fusion as described in claim 7, characterized in that: The method for generating the meta-gradient signal by the optimization module includes: An approximate estimate of the impact of single-sample annotation on the performance of the main task model on the validation set is constructed, and the directional consistency between the overall loss gradient of the validation set and the loss gradient of a single sample is used as a measure of the expected contribution of the sample; the higher the directional consistency, the greater the expected improvement in the generalization performance of the model after annotating the current sample. Using the sample selection probability distribution output by the discrimination module as weights, the expected contributions of all unlabeled samples are weighted and summed to construct a differentiable substitution loss function. Batch sampling without replacement is decomposed into multiple sequential single-sample sampling steps. In each step, a selection result is generated based on the currently unselected sample and its current selection probability distribution. The selection result retains the characteristic of being differentiable. In the forward propagation phase, discrete sampling results are used to participate in the main task model update, and in the back propagation phase, continuous gradient information is used to update parameters, so that the meta-gradient signal can be completely transmitted back through the sampling operation. The derivative of the substitution loss function with respect to the sample selection probability distribution is used to generate a meta-gradient signal; each component of the meta-gradient signal is used to indicate the adjustment of the selection probability of the corresponding sample: the selection probability of samples with large expected contributions should be increased, and the selection probability of samples with small expected contributions should be decreased. The meta-gradient signal is transmitted sequentially through the backpropagation path to the coordination network of the discrimination module, the policy agents of the policy module, and the feature encoding network of the feature module, driving the joint update of the parameters of each module.

10. An advanced active learning sampling method based on multi-policy fusion, comprising an advanced active learning system based on multi-policy fusion as described in any one of claims 1-9; characterized in that: S1. Extract features from the original samples in the unlabeled sample pool using a feature encoding network to obtain sample feature representations; S2. Input the sample feature representation into multiple different policy agents; each policy agent evaluates the information content of the sample in parallel based on different underlying objectives, and generates its own policy score vector; the underlying objectives include evaluating the classification uncertainty, feature space diversity and main task model influence of the sample; S3. Obtain the policy score vectors output by all policy agents and the current main task model state; dynamically evaluate and fuse the policy score vectors through a coordination network to predict the improvement of the main task model performance by different sample selection schemes, and output the sample selection probability distribution; select a batch of samples to be labeled from the unlabeled sample pool using a random sampling method without replacement based on the distribution, and renormalize the probability distribution of the remaining samples after sampling. S4. The selected sample batches are labeled and used to update the main task model; based on the performance improvement of the updated main task model on the validation set, a meta-gradient signal is generated; the meta-gradient signal is backpropagated through a differentiable path to jointly optimize the learnable parameters of the coordination network, all policy agents, and the feature encoding network.

Images