A classification method and system based on category-aware attention

Abstract

The application discloses a classification method and system based on category-aware attention, and belongs to the fields of machine learning and data science. The method comprises the following steps: inputting a test instance after normalization into a classification model composed of multiple base classifiers to obtain confidence scores of the test instance belonging to different categories; wherein, after embedding the test instance into a latent space, the classification model uses a category attention head to map the embedding representation of the test instance to a category-specific space of each category, and combines the attention weights of all base classifiers in the category-specific space to obtain the confidence score of the test instance belonging to the category; and obtaining a classification result of the test instance based on the confidence scores of the test instance belonging to different categories. The application can more accurately classify data.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and data science, and in particular to a classification method and system based on category-aware attention. Technical Background

[0002] In the field of machine learning, ensemble learning methods, by integrating the prediction results of multiple base learners, improve the accuracy and stability of models and have become an important tool with wide applications. Traditional ensemble methods such as Bagging and Boosting typically assume that all base learners have consistent prediction performance across all instances. However, in practical applications, different base learners may exhibit significant differences in performance across different data regions. Therefore, to address this issue, some Dynamic Ensemble Selection (DES) models introduce k-nearest neighbor information, dynamically selecting a suitable set of classifiers for each instance.

[0003] In existing DES models, the META-DES framework utilizes meta-learning to evaluate the capabilities of base classifiers through multiple meta-feature sets (such as local accuracy, confidence, and output features), providing a comprehensive perspective on classifier performance. Furthermore, methods such as KNORA-Eliminate (KNORA-E) and KNORA-Union (KNORA-U) improve ensemble performance by selecting a completely accurate classifier within the capability region, or by selecting a classifier capable of correctly classifying at least one sample. While these methods enhance model robustness and accuracy to some extent, they rely on the implementation of k-nearest neighbors, meaning that k nearest neighbors need to be searched in the training dataset for each test instance, resulting in excessively high computational complexity and making them unsuitable for real-world applications.

[0004] To address this issue, researchers proposed the Attentional Ensemble Model (AEM), which learns personalized classifier weights for each instance through an attention mechanism, thereby improving the performance of multi-classifier systems. AEM also introduces a diversity measure to ensure the diversity of the classifier set, thus enhancing the model's robustness and accuracy. Furthermore, Mixture of Expert (MoE) systems solve complex tasks by dynamically selecting and combining the outputs of expert models and introduce sparse gating mechanisms to reduce computational complexity. However, these global model-based methods typically assume that instances of the same class are centrally distributed in the feature space, ignoring the fact that instances of the same class may be distributed across multiple regions.

[0005] In summary, existing dynamic ensemble selection methods and global model-based ensemble learning methods have limitations in handling instance distribution complexity, making it difficult to effectively identify all instances, especially in terms of computational complexity and the accuracy of classifier selection, which still need improvement. Summary of the Invention

[0006] To overcome the limitations of existing technologies, this invention proposes a classification method and system based on category-aware attention, which can classify data more accurately.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] A classification method based on category-aware attention, the method comprising:

[0009] After normalization, the test instance is input into a classification model composed of multiple base classifiers to obtain the confidence score of the test instance belonging to different categories. The classification model embeds the test instance into the latent space, maps the embedding representation of the test instance to each category-specific space using a category attention head, and combines the attention weights of all base classifiers in the category-specific space to obtain the confidence score of the test instance belonging to that category.

[0010] Based on the confidence scores of the test instance belonging to different categories, the classification result of the test instance is obtained.

[0011] Further, training the classification model includes:

[0012] The instances and base classifiers are embedded into the latent space to obtain the embedded representations of the instances and base classifiers;

[0013] By using the category attention head, the embedding representations of instances and base classifiers are mapped to a specific space of each category, resulting in the instance embedding representation and base classifier embedding representation of that category.

[0014] Based on the instance embedding representation and base classifier embedding representation of the category, calculate the attention weight of each base classifier in the category;

[0015] By combining the attention weights of the base classifier and the prediction results of the base classifier for the instance, the confidence score of the instance belonging to the category is obtained;

[0016] In each training cycle, the training loss function is... L1 represents the attention loss function used to enhance attention to base classifiers that correctly classify instances and reduce reliance on base classifiers that misclassify instances; L2 represents the hinge loss function used to avoid the model from over-relying on a single base classifier. LCLThis represents the local contrastive loss function used to increase the likelihood of sampling hard negatives. Let λ1 represent the weights assigned to a given instance, λ2 represent the weights assigned to the hinge loss function L2, and λ3 represent the weights assigned to the local contrast loss function L2. LCL The weight given.

[0017] Furthermore, before embedding the instances and base classifiers into the latent space to obtain the embedded representations of the instances and base classifiers, the method further includes:

[0018] Validate all base classifiers on the validation set;

[0019] If the base classifier c j The classification result of a set of instances is completely determined by another base classifier c. k If the base classifier c is covered, then it will be filtered out. j .

[0020] Furthermore, the embedding of instances into the latent space to obtain the embedded representation of the instances includes:

[0021] For numerical features, normalization is performed;

[0022] and,

[0023] For categorical features, convert the categorical feature into a one-hot vector.

[0024] Furthermore, the embedding of the base classifier into the latent space to obtain the embedded representation of the base classifier includes:

[0025] One-hot encoding is performed on the base classifiers according to their numbers to obtain one-hot vectors;

[0026] Initialize an embedding matrix of dimension N×M, where N represents the dimension of the base classifier embeddings and M is the number of base classifiers; wherein, the embedding matrix is ​​updated as the classification model learns;

[0027] Multiplying the embedding matrix with the one-hot vector of the base classifier yields the embedding representation of the base classifier.

[0028] Further, based on the instance embedding representation and base classifier embedding representation of the category, the attention weight of each base classifier in the category is calculated, including:

[0029] Attention values ​​are calculated based on instance embedding representations and base classifier embedding representations. in, e represents the embedding representation of instance i. j Let h represent the embedding representation of the base classifier j, and y represent the class. iThe label representing instance i, Let Q represent the first transformation matrix for category h, and let Q represent the query in the attention mechanism. Let K represent the second transformation matrix for category h, and K represent the key in the attention mechanism. Indicates based on the first transformation matrix Embedded representation Perform the transformation, where d represents the embedded representation. and embedding representation e j The vector dimension, τ represents the scaling factor used to adjust the sharpness of the attention distribution;

[0030] Calculate the attention weight of instance i in class h for base classifier j. Where M is the number of base classifiers.

[0031] Furthermore, the attention loss function This represents the attention weight of instance i in class h for base classifier j. It is an indicator function; the hinge loss function m2 represents the second preset threshold; the local contrast loss function m1 represents the first set threshold, dist represents the distance function, and q i The embedded representation of instance i, This represents the embedding representation corresponding to the positive instance of instance i. This represents the embedding representation corresponding to the negative instance of instance i.

[0032] Further, the classification result of the test instance based on the confidence scores of the test instance belonging to different categories includes:

[0033] Obtain the third set threshold m3 and the fourth set threshold m4;

[0034] Select the highest confidence score F from the confidence scores of different categories. h Where h represents the category;

[0035] If the confidence score F h The difference between the confidence score and any other confidence score is greater than the third preset threshold m3, and the confidence score F h If the value is greater than the fourth set threshold m4, then category h is taken as the classification result of the test instance;

[0036] If the confidence score F h The difference between the confidence score and any other confidence score is less than the third preset threshold m3, or the confidence score F h If the value is less than the fourth preset threshold m4, then obtain the confidence score F.h The confidence scores of the test instance are less than the third set threshold m3, forming a category candidate set. Based on the number of instances similar to the test instance in the category space, a category is selected from the category candidate set as the classification result of the test instance.

[0037] Further, the step of selecting a category as the classification result of the test instance from the category candidate set based on the number of instances similar to the test instance in the category space includes:

[0038] The k-nearest neighbors algorithm is used to find instances similar to the test instance in the corresponding class space to construct several sets.

[0039] Statistics for each set The number of instances is determined, and the category with the most instances is used as the classification result for the test instance.

[0040] A classification system based on category-aware attention, characterized in that the system comprises:

[0041] The confidence calculation module is used to input the normalized test instance into a classification model composed of multiple base classifiers to obtain the confidence score of the test instance belonging to different categories. The classification model embeds the test instance into the latent space, maps the embedding representation of the test instance to each category-specific space using a category attention head, and combines the attention weights of all base classifiers in the category-specific space to obtain the confidence score of the test instance belonging to that category.

[0042] The classification result generation module is used to obtain the classification result of the test instance based on the confidence scores of the test instance belonging to different categories.

[0043] Compared to existing technologies, this invention fully leverages the advantages of global models and local information, adaptively adjusting the ensemble weights of each base classifier based on the characteristics of each instance. In this method, the classification task for each category is treated as a sub-task, and a category-aware attention mechanism is designed to capture the features of each category and learn the predictive capabilities of each classifier across different categories. Furthermore, this invention introduces a local contrastive loss module, encouraging the model to learn representations that capture local similarities, thereby enhancing the model's ability to discern fine-grained patterns in the data. Attached Figure Description

[0044] Figure 1 This is a flowchart of a category-aware attention-based classification method.

[0045] Figure 2 This is a flowchart of the model training process.

[0046] Figure 3 This is a flowchart of the model during the generalization phase.

[0047] Figure 4 This is a first-result graph showing the model's performance compared to existing models on common public datasets.

[0048] Figure 5 This is a second result graph showing the performance of the model and existing models on common public datasets.

[0049] Figure 6 This section describes the types of base classifiers used in the model. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] The classification method based on category-aware attention of the present invention, such as Figure 1 As shown, it includes the following steps.

[0052] Step 1: After normalizing the test instance, input it into a classification model composed of multiple base classifiers to obtain the confidence score of the test instance belonging to different categories.

[0053] The category-aware attention-based classification method of this invention is implemented based on a classification model. The training flowchart of this classification model is as follows: Figure 2 As shown, the process includes: removing redundant base classifiers based on their performance on the training and validation sets; normalizing the features of instances for the retained base classifiers; inputting the preprocessed data into the category-aware attention mechanism module to map instances to different category spaces and calculate the confidence score of each instance in different categories; finding the most similar instances to form a positive sample set for each instance and randomly sampling to obtain a negative sample set, using a local contrastive loss module to enhance representation learning; and calculating the attention loss, hinge loss, and weight of each sample, combined with the local contrastive loss, to calculate the final loss function for optimizing model parameters. For a detailed description of the training process of this classification model, please refer to the following steps.

[0054] Step 1.1: Eliminate redundant base classifiers based on their performance on the training and validation sets.

[0055] To address the problem of redundant base classifiers, the training device first filters the base classifiers before embedding them into the decision space. The purpose of this filtering process is to ensure diversity among the base classifiers and avoid redundant ones. During the filtering process, if a certain base classifier c... j The classification result of a set of instances is completely determined by another base classifier c.k If covered, then c is considered to be covered. j Redundant classes are removed from the base classifier set. This filtering process is performed on the validation set to prevent overfitting.

[0056] Step 1.2: Normalize the features of the instances for the retained base classifiers.

[0057] This invention obtains embedded representations of instances and base classifiers that achieve feature normalization by embedding instances and base classifiers into a latent space. Specifically, the training device in this embodiment acquires a sample set, including sample data and sample annotations, data acquisition, and preprocessing. First, instances and base classifiers are embedded into the latent space. Specifically, the numerical features of instances are normalized, and categorical features are converted into one-hot codes to obtain an encoded representation for each instance. This ensures that in practical applications, all numerical features are appropriately scaled, and categorical features are effectively represented. After obtaining the encoded representation, the training device combines the feature space of the instances with the decision space of the base classifiers to obtain an embedded representation for each instance. Furthermore, the base classifiers are screened to ensure diversity and avoid redundancy, and the screened base classifiers are represented as corresponding embedding vectors. Through these steps, the accuracy and robustness of classification can be improved.

[0058] Specifically, the training device first embeds instances and base classifiers, mapping them into the latent space. To do this, the features of the instances must first be encoded. Numerical features are normalized; categorical features are converted into one-hot vectors. This method yields the encoded representation of each instance. This process ensures that numerical features are appropriately scaled, while categorical features are effectively represented through one-hot encoding. Finally, instance x... i It is represented as a vector of length L in the feature space.

[0059] For each base classifier c that is retained j It can be achieved through the corresponding embedding vector e jThis vector is obtained by multiplying a one-hot vector by the embedding matrix H. The one-hot vector is generated based on the base classifier numbers. For example, if there are 5 base classifiers, the one-hot vector of the first base classifier is [1,0,0,0,0], the one-hot vector of the second base classifier is [0,1,0,0,0], and so on. The embedding matrix contains the embedding representations of all base classifiers, ensuring the unique contribution of each base classifier in the decision space. The rows and columns of the embedding matrix represent the dimension of the base classifier embedding vector and the number of base classifiers, respectively. Specifically, this invention first randomly initializes the embedding matrix H. Then, as the model trains, the embedding representation of each base classifier (corresponding to a column in the embedding matrix H) is learned and updated, resulting in the trained embedding.

[0060] Through the above steps, the features of the instance and the decision information of the base classifier are effectively embedded into the latent space, providing a more accurate and robust foundation for subsequent classification and prediction.

[0061] Step 1.3: Input the preprocessed data into the category-aware attention mechanism module, map the instances to the space of different categories, and calculate the confidence score of each instance in different categories.

[0062] This invention integrates base classifiers and instances embedded in a latent space using a category-aware attention mechanism for output. Instances of different categories exhibit significant differences in distribution; therefore, traditional multi-head attention mechanisms, applied indiscriminately, may fail to capture the unique features of each category, thus affecting the model's prediction accuracy. To address this, this solution treats each category classification as an independent task and uses a corresponding attention head for each category, mapping the embedded representations of instances and base classifiers to a category-specific space. Then, a scaled dot-product attention mechanism is used to calculate the predictive power of each base classifier for instances of that category. Finally, the prediction results of each base classifier are weighted and summed to obtain the confidence score of each instance in its corresponding category space. By introducing a scaling parameter, the concentration of attention distribution can be effectively adjusted, allowing the model to better focus on key information relevant to specific instances. Based on the above, the present invention realizes the use of category attention heads to map the embedding representations of instances and base classifiers to a specific space of each category, thereby obtaining the instance embedding representation and base classifier embedding representation of the category, calculating the attention weight of base classifier j for instance i in category h, and combining the attention weight of the base classifier and the prediction result of the base classifier for the instance to obtain the confidence score of the instance belonging to the category.

[0063] Specifically, to calculate the category of each instance, a category-aware attention mechanism is designed for the training device. The model learns the relationship between base classifiers and instances by embedding them into a latent space using the attention mechanism. Traditional multi-head attention mechanisms may fail to fully capture the unique features of each category when dealing with instances of different categories, leading to a decrease in classification accuracy. To address this issue, this invention draws on the idea of ​​expert hybrid models, treating the classification task for each category as an independent task. Specifically, for each category, the model uses a corresponding attention head to map the embedding vectors of instances and base classifiers into a category-specific space. In this way, the model can construct independent representations for each category and further calculate category-based attention weights. The attention weights reflect the predictive power of a base classifier on a specific category and are calculated using a scaled dot product attention mechanism.

[0064]

[0065] in, e represents the embedding representation of instance i. j Let h represent the embedding representation of the base classifier j, and y represent the class. i The label representing instance i, Let Q represent the first transformation matrix for category h, and let Q represent the query in the attention mechanism. Let K represent the second transformation matrix for category h, and K represent the key in the attention mechanism. Indicates based on the first transformation matrix Embedded representation Perform the transformation, where d represents the embedded representation. and embedding representation e j The vector dimension is τ, which represents a scaling factor used to adjust the sharpness of the attention distribution. By adjusting the scaling factor τ, the attention weights can be concentrated more on a few key elements or distributed more evenly. This mechanism helps the model better focus on important information relevant to a specific instance, thereby improving classification accuracy and robustness.

[0066] Finally, the model sums the outputs of each base classifier according to attention weights to generate a confidence score for each instance in each class. This design ensures that the model can make accurate predictions based on the characteristics of each class when faced with instances of different classes, effectively improving the overall performance of the classifier.

[0067] Step 1.4: For each instance, find the most similar instances to form a positive sample set, and randomly sample to obtain a negative sample set. Use the local contrastive loss module to enhance representation learning.

[0068] This invention enhances representation learning by introducing a local contrastive loss module. Since instances of different categories typically exhibit significantly different distributions, and even within the same category, the distributions of instances can vary considerably, simply relying on category representations may not effectively distinguish instances within the same category distributed across different regions. Therefore, this technical solution introduces a local contrastive loss module to improve the accuracy of representation learning. Specifically, the K-nearest neighbor algorithm is used to find the most similar instances for each instance, forming a positive sample set. This ensures that positive samples not only belong to the same category but also possess similar features, thereby prompting the model to learn and capture representations of local similarity. After acquiring positive and negative samples, the local contrastive loss is calculated. By adjusting the relative distance between positive and negative samples, the model's ability to recognize fine-grained patterns is further improved, thus significantly enhancing overall classification performance.

[0069] Specifically, to improve the performance of representation learning, especially when dealing with inconsistent instance distribution within a category, this invention introduces local contrastive loss. Instances of different categories often have significantly different distribution characteristics, but even within the same category, the distribution of instances can vary considerably. For instances scattered across multiple regions, relying solely on category representations is insufficient for effective differentiation, especially among instances distributed across different regions within the same category. To address this issue, this module utilizes the K-nearest neighbor algorithm to find the K most similar positive instances for each instance. These positive instances belong to the same category as the current instance and have similar features in the feature space. In this way, the model is encouraged to learn representations that capture local similarity, thereby improving its ability to capture fine-grained patterns in the data. This positive sample selection method ensures that positive instances not only belong to the same category but are also highly similar in features. For negative example selection, the local contrastive loss module samples instances from different categories that are difficult to classify as negative examples, based on the classification error of the base classifier. This method increases the likelihood of sampling "hard negative examples," effectively improving the model's contrastive learning performance. In each training cycle, the model randomly selects one positive example and one negative example for comparative learning:

[0070]

[0071] Where m1 represents the first set threshold, q i , and This represents the embedded representation after corresponding category attention head mapping. This loss function measures the distance difference between the current instance and both positive and negative examples, optimizing the model's learning process by adjusting the relative magnitude of the distances. The introduced boundary parameters control the difference between the distance between the current instance and positive examples relative to the distance with negative examples, ensuring the model learns features that distinguish between positive and negative samples more effectively.

[0072] Step 1.5: Calculate attention loss and hinge loss.

[0073] To further improve classification accuracy, this invention also uses attention loss function and hinge loss function during training to adjust the parameters of the classification model and obtain a trained classification model.

[0074] The L1 attention loss function aims to increase attention to base classifiers that correctly classify instance i, while reducing attention to misclassifying base classifiers. This L1 attention loss function is designed to allow the model to gradually learn during training which base classifiers perform better in classifying specific instances, thus enabling it to more effectively allocate attention to those accurate classifiers during the generalization phase. This approach ensures that the model prioritizes accurate base classifiers during generalization, thereby improving classification performance.

[0075]

[0076] in, The attention weights represent the ability of base classifier j to predict instance i in class h. It is an indicator function, which indicates the prediction of the label v of instance i by the base classifier j. ij With real label y i If the values ​​are the same, the value is 1; otherwise, it is 0.

[0077] The hinge loss function L2 is designed to prevent excessive focus on a single base classifier, which could undermine the advantages of ensemble learning and lead to overfitting. Therefore, this invention introduces another hinge loss function, L2, to penalize this situation. This L2 loss function sets a threshold value; when the attention score exceeds this threshold, a penalty is applied, thereby preventing the model from over-relying on a single base classifier and reducing the risk of overfitting.

[0078]

[0079] in, is the maximum attention weight of instance i under category h, and m2 is the second set threshold. Exceeding this threshold m2 will result in a penalty.

[0080] Step 1.6: Calculate the weight of each sample, and calculate the final loss function based on local contrast loss, attention loss, and hinge loss to optimize the model parameters.

[0081] To address the issue of varying instance importance during the training phase, the training device also assigns a weight to each instance i. This is to assign higher weights to instances with higher classification error rates, encouraging the model to pay more attention to these error-prone instances and thus reflecting the importance of instances in the learning process. Here, M′ represents the number of base classifiers, 1(c j (x i )==y i ) is an indicator function. Its value is 1 when the predicted label of the base classifier j for instance i is the same as the true label, and 0 otherwise.

[0082] To balance the weight differences caused by the number of instances in different categories, this invention also introduces category weights. Category weight Adjustments are made based on the number of instances in each category to ensure that instances in each category receive appropriate attention during the model learning process.

[0083]

[0084] Where, n j This represents the number of instances of category j. It is the total number of categories. It is the number of instances of the category to which instance i belongs.

[0085] The final instance weight is

[0086] To effectively learn all parameters, this invention defines an objective function L. This objective function L comprehensively considers the various loss functions mentioned above and optimizes the overall performance of the model by adjusting the weight coefficients of different losses. To achieve this goal, the training device uses the Adam optimization algorithm to optimize the objective function. In the implementation of the algorithm, an initialization step is first performed, randomly setting the initial parameters of the model. Then, an iterative training process is entered, gradually optimizing the model parameters until convergence.

[0087]

[0088] λ1 and λ2 are used to control the weights of different losses. This invention optimizes the objective function using the Adam optimizer.

[0089] In one embodiment, after training the prototype network using the objective function, the training device can also output the trained prototype network to the execution device (e.g., a terminal device, an edge device, or other target device). In the above embodiments of this application, it is specifically described that the prototype network trained by the training device can be sent to the execution device, where the execution device can specifically apply the trained prototype network. This separates the training and application processes of the prototype network, reducing the computational load on the execution device and saving its computing power.

[0090] Step 2: Based on the confidence scores of the test instance belonging to different categories, obtain the classification result of the test instance.

[0091] Related devices (such as execution devices or training devices) can make predictions using this trained prototype network. The execution device can utilize a novel model application approach. In the application phase, the execution device first applies a category-aware attention mechanism from a global perspective for prediction. This step obtains the confidence scores of each instance across all categories by applying all attention heads to each instance, and then selects the category with the highest confidence score as the predicted label. However, relying solely on global information can lead to classification difficulties for some instances because the probability values ​​of certain categories may be very close. In such cases, the execution device needs to consider local information. Specifically, when the class rankings are very close, or when the confidence scores of all categories are low after each attention head transformation, the execution device introduces local information to assist in classification.

[0092] The execution device transforms all instances in the training set according to their respective categories and uses the k-nearest neighbor algorithm to find the set of instances closest to the current instance. Then, the execution device counts the number of instances belonging to each category in these sets and finally assigns the current instance to the category with the most instances. This method effectively utilizes local information to compensate for the lack of global information, improving classification accuracy.

[0093] Specifically Figure 3 This is a flowchart of the model during the generalization phase. During classification, the execution device first applies a category-aware attention mechanism from a global perspective to predict each instance. Specifically, the execution device transforms instance i through attention heads for all categories, generating a confidence score for each category. Then, the execution device selects the category with the highest confidence score as the predicted label for instance i.

[0094] However, relying solely on global information can lead to difficulties in classifying certain instances, especially when the probability values ​​(confidence scores) of some classes are very close. To address this issue, the execution device also needs to consider local information. For example, when the probability values ​​of two classes are not significantly different and the transformed confidence scores are low, the execution device will consider the local information of instance i. Specifically, the execution device uses attention heads for classes k1 and k2 to transform all instances in the training set, obtaining their representations in their respective class spaces. Subsequently, instance i also undergoes a corresponding transformation to obtain its representation q in the class k1 and k2 spaces. i,k1 and q i,k2 Next, the execution device uses the k-nearest neighbor algorithm to find the N instances most similar to instance i in the corresponding class space, denoted as Ni, ... and

[0095]

[0096] Among these similar instances, the execution device counts the number of instances belonging to categories k1 and k2, and assigns instance i to the category with more similar instances. If the number of similar instances in category k1 is greater than or equal to the number of similar instances in category k2, then instance i is classified as k1; otherwise, it is classified as k2. By combining global and local information, the execution device can classify instances more accurately during the prediction inference phase.

[0097] This invention provides a machine learning model training method based on a category-aware attention mechanism and local contrastive loss. By setting different category-specific attention heads and combining global and local information, the training device can more accurately capture the features of various types of data, thereby improving classification accuracy and robustness. By introducing a local contrastive loss module, the training device can effectively distinguish instances with large distribution differences within the same category, improving the model's performance in complex scenarios. Because an optimization strategy based on instance weights and category weights is used during training, the execution device can better adapt to different task scenarios during the inference phase and effectively avoid overfitting. Furthermore, the method of this invention also adjusts the parameters of the attention distribution, enabling the training device to flexibly handle datasets of varying complexity, further enhancing its adaptability and generalization ability.

[0098] In summary, when using the training method provided by this invention, the obtained model significantly outperforms traditional methods on multiple tasks, and the experimental results strongly demonstrate the effectiveness and reliability of the training method provided by this invention. The method of this invention is highly adaptable and can be applied to various machine learning scenarios and applications. The method of this invention can effectively achieve the purpose of automatically selecting and optimizing machine learning models; it is intuitive, effective, and easy to use.

[0099] Figure 4 and Figure 5 This figure compares the experimental results of the method proposed in this invention with existing ensemble learning methods. The datasets selected for this invention cover several domains, including financial credit risk analysis, disease analysis and diagnosis, medical disease classification, observation signal classification, defect detection, population health and nutrition classification, etc. The method proposed in this invention demonstrates superior classification accuracy and lower variance in 95% of the datasets.

[0100] It should be noted that the ensemble learning method involved in this invention does not have explicit requirements for the base classifier, such as... Figure 6 As shown, it can be a common machine learning algorithm (such as decision tree, logistic regression, support vector machine, etc.) or a neural network.

[0101] Finally, the method of this invention is merely a preferred embodiment and is not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A classification method based on category-aware attention, characterized in that, The method includes: After normalization, the test instance is input into a classification model composed of multiple base classifiers to obtain the confidence score of the test instance belonging to different categories. The classification model embeds the test instance into the latent space, maps the embedding representation of the test instance to each category-specific space using a category attention head, and combines the attention weights of all base classifiers in the category-specific space to obtain the confidence score of the test instance belonging to that category. Based on the confidence scores of the test instance belonging to different categories, the classification result of the test instance is obtained; Training the classification model includes: The instances and base classifiers are embedded into the latent space to obtain the embedded representations of the instances and base classifiers; By using the category attention head, the embedding representations of instances and base classifiers are mapped to a specific space of each category, resulting in the instance embedding representation and base classifier embedding representation of that category. Based on the instance embedding representation and base classifier embedding representation of the category, calculate the attention weight of each base classifier in the category; By combining the attention weights of the base classifier and the prediction results of the base classifier for the instance, the confidence score of the instance belonging to the category is obtained; In each training cycle, the training loss function is... L1 represents the attention loss function used to enhance attention to base classifiers that correctly classify instances and reduce reliance on base classifiers that misclassify instances; L2 represents the hinge loss function used to avoid the model from over-relying on a single base classifier. LCL This represents the local contrastive loss function used to increase the likelihood of sampling hard negatives. Let λ1 represent the weights assigned to a given instance, λ2 represent the weights assigned to the hinge loss function L2, and λ3 represent the weights assigned to the local contrast loss function L2. LCL Weighting; The attention loss function This represents the attention weight of instance i in class h for base classifier j. It is an indicator function; The hinge loss function m2 represents the second set threshold. This represents the attention weight of instance i in category h; The local contrast loss function m1 represents the first set threshold, dist represents the distance function, and q i The embedded representation of instance i, This represents the embedding representation corresponding to the positive instance of instance i. The embedding representation corresponding to the negative instance of instance i; The attention weights of each base classifier in the category are calculated based on the instance embedding representation and base classifier embedding representation of that category, including: Attention values ​​are calculated based on instance embedding representations and base classifier embedding representations. in, e represents the embedding representation of instance i. j Let h represent the embedding representation of the base classifier j, and y represent the class. i The label representing instance i, Let Q represent the first transformation matrix for category h, and let Q represent the query in the attention mechanism. Let K represent the second transformation matrix for category h, and K represent the key in the attention mechanism. Indicates based on the first transformation matrix Embedded representation Perform the transformation, where d represents the embedded representation. and embedding representation e j The vector dimension, τ represents the scaling factor used to adjust the sharpness of the attention distribution; Calculate the attention weight of instance i in class h for base classifier j. Where M is the number of base classifiers.

2. The method according to claim 1, characterized in that, Before embedding instances and base classifiers into the latent space to obtain embedded representations of instances and base classifiers, the method further includes: Validate all base classifiers on the validation set; If the base classifier c j The classification result of a set of instances is completely determined by another base classifier c. k If the base classifier c is covered, then it will be filtered out. j .

3. The method according to claim 1, characterized in that, The step of embedding instances into the latent space to obtain the embedded representation of the instances includes: For numerical features, normalization is performed; and, For categorical features, convert the categorical feature into a one-hot vector.

4. The method according to claim 1, characterized in that, The base classifiers are embedded into the latent space to obtain the embedded representations of the base classifiers, including: One-hot encoding is performed on the base classifiers according to their numbers to obtain one-hot vectors; Initialize an embedding matrix of dimension N×M, where N represents the dimension of the base classifier embeddings and M is the number of base classifiers; wherein, the embedding matrix is ​​updated as the classification model learns; Multiplying the embedding matrix with the one-hot vector of the base classifier yields the embedding representation of the base classifier.

5. The method according to claim 1, characterized in that, The classification result of the test instance is obtained based on the confidence scores of the test instance belonging to different categories, including: Obtain the third set threshold m3 and the fourth set threshold m4; Select the highest confidence score F from the confidence scores of different categories. h Where h represents the category; If the confidence score F h The difference between the confidence score and any other confidence score is greater than the third preset threshold m3, and the confidence score F h If the value is greater than the fourth set threshold m4, then category h is taken as the classification result of the test instance; If the confidence score F h The difference between the confidence score and any other confidence score is less than the third preset threshold m3, or the confidence score F h If the value is less than the fourth preset threshold m4, then obtain the confidence score F. h The confidence scores of the samples whose difference is less than the third set threshold m3 are used to form a category candidate set. Based on the number of instances in the category space that are similar to the test instance, a category is selected from the category candidate set as the classification result of the test instance.

6. The method according to claim 5, characterized in that, The step of selecting a category as the classification result of the test instance from the category candidate set based on the number of instances similar to the test instance in the category space includes: The k-nearest neighbors algorithm is used to find instances similar to the test instance in the corresponding class space to construct several sets. Statistics for each set The number of instances is determined, and the category with the most instances is used as the classification result for the test instance.

7. A classification system based on category-aware attention, characterized in that, The system includes: The confidence calculation module is used to input the normalized test instance into a classification model composed of multiple base classifiers to obtain the confidence score of the test instance belonging to different categories. The classification model embeds the test instance into the latent space, maps the embedding representation of the test instance to each category-specific space using a category attention head, and combines the attention weights of all base classifiers in the category-specific space to obtain the confidence score of the test instance belonging to that category. The classification result generation module is used to obtain the classification result of the test instance based on the confidence scores of the test instance belonging to different categories; Training the classification model includes: The instances and base classifiers are embedded into the latent space to obtain the embedded representations of the instances and base classifiers; By using the category attention head, the embedding representations of instances and base classifiers are mapped to a specific space of each category, resulting in the instance embedding representation and base classifier embedding representation of that category. Based on the instance embedding representation and base classifier embedding representation of the category, calculate the attention weight of each base classifier in the category; By combining the attention weights of the base classifier and the prediction results of the base classifier for the instance, the confidence score of the instance belonging to the category is obtained; In each training cycle, the training loss function is... L1 represents the attention loss function used to enhance attention to base classifiers that correctly classify instances and reduce reliance on base classifiers that misclassify instances; L2 represents the hinge loss function used to avoid the model from over-relying on a single base classifier. LCL This represents the local contrastive loss function used to increase the likelihood of sampling hard negatives. Let λ1 represent the weights assigned to a given instance, λ2 represent the weights assigned to the hinge loss function L2, and λ3 represent the weights assigned to the local contrast loss function L2. LCL Weighting; The attention loss function This represents the attention weight of instance i in class h for base classifier j. It is an indicator function; The hinge loss function m2 represents the second set threshold. This represents the attention weight of instance i in category h; The local contrast loss function m1 represents the first set threshold, dist represents the distance function, and q i The embedded representation of instance i, This represents the embedding representation corresponding to the positive instance of instance i. The embedding representation corresponding to the negative instance of instance i; The attention weights of each base classifier in the category are calculated based on the instance embedding representation and base classifier embedding representation of that category, including: Attention values ​​are calculated based on instance embedding representations and base classifier embedding representations. in, e represents the embedding representation of instance i. j Let h represent the embedding representation of the base classifier j, and y represent the class. i The label representing instance i, Let Q represent the first transformation matrix for category h, and let Q represent the query in the attention mechanism. Let K represent the second transformation matrix for category h, and K represent the key in the attention mechanism. Indicates based on the first transformation matrix Embedded representation Perform the transformation, where d represents the embedded representation. and embedding representation e j The vector dimension, τ represents the scaling factor used to adjust the sharpness of the attention distribution; Calculate the attention weight of instance i in class h for base classifier j. Where M is the number of base classifiers.

Images