Cross-language few-sample classification method based on instance collaboration

Abstract

The embodiment of the invention provides a cross-language few-sample classification method based on instance collaboration. The method is applied to the technical field of natural language processing. Generating a pseudo tag for each sample in the target language data set through a pre-trained source language model and an initialized target language model; determining the reliability weight of each sample pseudo tag in the target language unlabeled data set, and constructing a weighted cross entropy loss function according to the reliability weight; iteratively updating the parameters of the target language model according to the weighted cross entropy loss, and stopping iteration until the performance index of the target language model on the target language verification set meets a preset condition or reaches the maximum number of iterations to obtain an optimized target language model; and inputting the to-be-classified text into the optimized target language model for analysis processing to obtain a classification result. According to the method, the generalization ability and the classification accuracy of low-resource languages are improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a cross-lingual few-shot classification method based on instance collaboration. Background Technology

[0002] Cross-lingual few-shot classification is a key technology in natural language processing, aiming to address the scarcity of labeled data in target languages ​​(especially low-resource languages) by leveraging the rich labeled data in source languages ​​(such as English), thereby achieving efficient knowledge transfer in tasks such as machine translation, social media analysis, and multilingual content moderation. However, this technology faces multiple challenges: First, lexical, grammatical, and cultural differences between languages ​​lead to distribution shifts, exacerbating data sparsity; second, label noise introduced during automatic labeling of target language data significantly reduces the model's generalization ability; and finally, under few-shot conditions, models are prone to overfitting and struggle to adapt to complex linguistic expressions in real-world scenarios.

[0003] Most existing cross-linguistic few-shot classification methods can be categorized into two types: translation-based strategies and cross-linguistic pre-trained model-based strategies.

[0004] Translation-based strategies rely on machine translation systems to convert source language-annotated data into the target language, and then process it using source language models. For example, semantic space mapping can be achieved through aligned cross-lingual word vectors or translation models (such as cross-lingual BERT). This method performs well when language pairs have high similarity (e.g., English-French), but its translation quality is poor for low-resource languages ​​(e.g., Swahili or Arabic), easily leading to semantic distortion and error propagation. Furthermore, the translation process cannot handle language-specific expressions (e.g., culture-specific metaphors) and heavily depends on the scalability and robustness of external translation tools.

[0005] Strategies based on cross-lingual pre-trained models: These methods leverage the shared representation space of multilingual pre-trained language models (such as XLM-R or mBERT) to directly transfer source language knowledge to the target language. These approaches reduce reliance on labeled target language data through few-shot fine-tuning or meta-learning (such as prototype networks). However, their performance is highly dependent on model alignment quality: when target language data is sparse or domain-specific, the model is susceptible to noisy labels, leading to overfitting; furthermore, pre-trained models fail to capture implicit semantic differences (such as lexical ambiguity), resulting in significantly reduced generalization ability in zero-shot or very few-shot scenarios.

[0006] The aforementioned methods all have significant limitations. Firstly, their data noise handling capabilities are insufficient: traditional denoising methods (such as confidence filtering or simple clustering) do not consider the semantic relationships between labels, failing to effectively clean up automatically generated pseudo-label data. Secondly, their transfer mechanisms are simplistic: existing methods largely rely on static feature alignment or shallow model collaboration, lacking dynamic modeling of instance-level semantic consistency and label dependencies, making it difficult for the models to adapt to complex linguistic phenomena ranging from explicit to implicit expressions. Furthermore, the lack of a progressive optimization mechanism prevents the models from iteratively optimizing the noise filtering process, limiting their practicality and scalability in low-resource scenarios. Summary of the Invention

[0007] This application provides a cross-lingual few-shot classification method based on instance collaboration to address the problems of poor automatic target language annotation quality, severe label noise interference, and insufficient adaptability to low-resource languages ​​in existing cross-lingual few-shot classification methods. The instance-collaboration-based cross-lingual few-shot classification method provided in this application includes:

[0008] Obtain the unlabeled dataset for the target language;

[0009] Pseudo-labels are generated for each sample in the target language dataset using a pre-trained source language model and an initialized target language model.

[0010] Determine the reliability weight of the pseudo-label for each sample in the unlabeled dataset of the target language, and construct a weighted cross-entropy loss function based on the reliability weight;

[0011] The parameters of the target language model are iteratively updated based on the weighted cross-entropy loss until the performance indicators of the target language model on the target language validation set meet the preset conditions or reach the maximum number of iterations. Then the iteration stops, and the optimized target language model is obtained.

[0012] The text to be classified is input into the optimized target language model for analysis and processing to obtain the classification result.

[0013] Optionally, determining the reliability weight of the pseudo-label for each sample in the unlabeled dataset of the target language includes:

[0014] The pre-trained language model encoder is used to encode each word in the unlabeled dataset of the target language to generate a word representation vector, and the representation vectors of all words are stored in the memory bank.

[0015] Nearest neighbor retrieval is performed in the memory based on the cosine similarity between the representation vectors to determine the nearest neighbor instances of each word;

[0016] Calculate the consistency score between each word pseudo-label and its nearest neighbor instance pseudo-labels;

[0017] The consistency score is converted into a reliability weight corresponding to each word pseudo-label.

[0018] Optionally, the cosine similarity satisfies the following formula:

[0019]

[0020] in, For words The representation vector, For words The representation vector, for The nearest neighbor instance.

[0021] Optionally, the consistency score between each word pseudo-label and its nearest neighbor instance pseudo-labels is calculated according to the following formula:

[0022]

[0023] in, For the source language model, words Predicted soft tags, For words Nearest neighbor instances, For words The number of nearest neighbor instances, For indicator functions, As a hard label, As a hard label, For the source language model, words Predicted soft labels.

[0024] Optionally, the consistency score is converted into a reliability weight corresponding to each word pseudo-label, satisfying the following formula:

[0025]

[0026] in, For hyperparameters, It refers to all predictions as categories in the unlabeled dataset of the target language. The average consistency score of the words.

[0027] Optionally, the weighted cross-entropy loss function satisfies the following formula:

[0028]

[0029] in, It is the sequence length. It is the cross-entropy loss function.

[0030] Optionally, the method further includes:

[0031] Pseudo-labels are generated for each sample in the unlabeled target language dataset using the optimized target language model;

[0032] A consistency check is performed between the pseudo-labels generated by the optimized target language model and the pseudo-labels generated by the pre-trained source language model, and samples with inconsistent pseudo-labels are re-labeled.

[0033] This application provides a cross-lingual few-shot classification method based on instance collaboration. The method involves acquiring an unlabeled dataset of the target language; generating pseudo-labels for each sample in the target language dataset using a pre-trained source language model and an initialized target language model; determining the reliability weights of the pseudo-labels for each sample in the unlabeled target language dataset and constructing a weighted cross-entropy loss function based on these weights; iteratively updating the parameters of the target language model based on the weighted cross-entropy loss until the performance metrics of the target language model on the target language validation set meet preset conditions or reach the maximum number of iterations, at which point iteration stops, resulting in an optimized target language model; and inputting the text to be classified into the optimized target language model for analysis to obtain the classification result. This method improves the generalization ability and classification accuracy for low-resource languages. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 A flowchart illustrating the cross-language few-shot classification method based on instance collaboration provided in this application embodiment;

[0036] Figure 2 A schematic diagram illustrating the process of iterative updating of the target language model provided in this application embodiment;

[0037] Figure 3 A schematic diagram of nearest neighbor retrieval provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the pseudo-label correction process provided in an embodiment of this application.

[0039] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0042] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0043] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0044] Figure 1 This is a flowchart illustrating the instance-based collaborative cross-lingual few-shot classification method provided in this embodiment. The instance-based collaborative cross-lingual few-shot classification method provided in this embodiment includes:

[0045] S1: Obtain the unlabeled dataset for the target language.

[0046] Among them, unlabeled target language datasets refer to datasets that contain only the original text or multimedia content of the target language, but have not been manually labeled (such as classification labels, entity labels, relation labels, etc.).

[0047] S2: Generate pseudo-labels for each sample in the target language dataset using the pre-trained source language model and the initialized target language model.

[0048] The source language model refers to a model pre-trained on a richly labeled source language, possessing cross-language generalization capabilities. An example of a pre-trained source language model is the XLM-R model trained on labeled English data. The initialization of the target language model can be random initialization or fine-tuning based on the source language model.

[0049] Specifically, for each sample in the unlabeled dataset of the target language, for example, for each word, the source language model and the target language model are input separately to obtain the prediction results of the two models. The prediction with high confidence is used as a pseudo label and paired with the original sample to form pseudo-labeled data.

[0050] S3: Determine the reliability weight of the pseudo-label for each sample in the unlabeled dataset of the target language, and construct a weighted cross-entropy loss function based on the reliability weight.

[0051] Specifically, a pre-trained language model encoder is used to encode each word in the unlabeled dataset of the target language to generate a word representation vector, and all word representation vectors are stored in a memory bank. The pre-trained language model encoder can be, for example, a multilingual BERT or XLM-R model, used to extract cross-lingual representation vectors, supporting multiple languages ​​including low-resource languages.

[0052] Furthermore, nearest neighbor retrieval is performed in the memory based on the cosine similarity between the representation vectors to determine the nearest neighbor instances of each word. Figure 3 This is a schematic diagram of nearest neighbor retrieval provided in an embodiment of this application. For example, the K nearest neighbor instances of each word are retrieved based on cosine similarity (K is usually set to 500), and the retrieval of the K nearest neighbor instances is implemented based on the FAISS library.

[0053] Specifically, the cosine similarity calculation satisfies the following formula:

[0054]

[0055] in, For words The representation vector, For words The representation vector, for The nearest neighbor instance.

[0056] Furthermore, the consistency score between each word pseudo-label and its nearest neighbor instance pseudo-labels is calculated, and the consistency score satisfies the following formula:

[0057]

[0058] in, For the source language model, words Predicted soft tags, For words Nearest neighbor instances, For words The number of nearest neighbor instances, This is an indicator function (returns 1 if the condition is true, otherwise returns 0). As a hard label, As a hard label, For the source language model, words Predicted soft labels. Consistency score. The higher the value, the more reliable the pseudo-tag.

[0059] Furthermore, the consistency score is converted into a reliability weight corresponding to each word pseudo-label, and the reliability weight satisfies the following formula:

[0060]

[0061] in, This is a hyperparameter (range [5,10], default value is 6) used to control the steepness of the weight distribution. It refers to all predictions as categories in the unlabeled dataset of the target language. The average consistency score (within-class mean) of the words. Obtained through online or pre-computation to ensure category-specific noise handling. These weights are used for soft filtering of samples during training, with high-weight samples contributing more to the loss function and low-weight samples (noise labels) having their contributions suppressed.

[0062] S4: Iteratively update the parameters of the target language model based on the weighted cross-entropy loss until the performance indicators of the target language model on the target language validation set meet the preset conditions or reach the maximum number of iterations, then stop the iteration and obtain the optimized target language model.

[0063] Figure 2 This is a schematic diagram illustrating the iterative update process of the target language model provided in this application embodiment. In each iteration: the source language model... and target language model For unlabeled datasets of the target language respectively After generating pseudo-labels and calculating reliability weights, the target language model is updated using a weighted loss function. ,in , The optimized hyperparameters utilize the updated target language model. Regenerate pseudo-labels and compare them with the source language model. The system performs consistency checks on the labels and corrects noisy labels. In each training iteration, the current input data is fed into the target language model. The model processes the input data based on its existing parameters and generates a predicted output. The predicted output is compared with the true label data, and the loss value under the current model parameters is calculated using the weighted cross-entropy loss function. Using the backpropagation algorithm, based on the calculated loss value, the model calculates along the gradient direction of the model parameters to determine the direction and magnitude of adjustment for each parameter. Based on the gradient information, the parameters of the target language model are iteratively updated, enabling the model to gradually reduce the loss value in subsequent training.

[0064] Specifically, the weighted cross-entropy loss function satisfies the following formula:

[0065]

[0066] in, It is the sequence length. This represents the cross-entropy loss.

[0067] If the performance improvement on the target language validation set stabilizes (e.g., the F1 score change is less than 0.5% for two consecutive rounds) or the maximum number of iterations is reached, the loop exits; otherwise, the next iteration continues. For example, the number of iterations is set to 2 to 5 rounds, and the model performance is evaluated using the target language validation set after each iteration. An early stopping strategy prevents overfitting.

[0068] In an optional embodiment, the method further includes: generating pseudo-labels for each sample in the unlabeled target language dataset using the optimized target language model; performing a consistency check on the pseudo-labels generated by the optimized target language model and the pseudo-labels generated by the pre-trained source language model, and relabeling samples with inconsistent pseudo-labels to correct noisy labels. The relabeling process is based on a voting mechanism, for example, retaining only samples where the two models predict the same result as high-confidence data. Figure 4 As shown, using the target language dataset For the target language model Training is performed, and the target language model is trained. Generating pseudo-labels using the source language dataset Source language model Training is performed, and the trained source language model is used. Generate pseudo-labels, relabel samples whose pseudo-labels generated by the two models are inconsistent, and repeat the above process.

[0069] S5: Input the text to be classified into the optimized target language model for analysis and processing to obtain the classification result.

[0070] Specifically, the optimized target language model is used to classify the text to be classified in the target language test set, and the classification results (such as entity labels or intent categories) are output. The evaluation metrics include micro-F1 score and accuracy.

[0071] The instance-based collaborative cross-lingual few-shot classification method provided in this application obtains an unlabeled dataset of the target language; generates pseudo-labels for each sample in the target language dataset using a pre-trained source language model and an initialized target language model; determines the reliability weight of the pseudo-labels for each sample in the unlabeled target language dataset, and constructs a weighted cross-entropy loss function based on the reliability weights; iteratively updates the parameters of the target language model according to the weighted cross-entropy loss until the performance indicators of the target language model on the target language validation set meet preset conditions or reach the maximum number of iterations, at which point iteration stops, resulting in an optimized target language model; the text to be classified is input into the optimized target language model for analysis and processing to obtain the classification result. This method improves the generalization ability and classification accuracy for low-resource languages.

[0072] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0073] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A cross-lingual few-shot classification method based on instance collaboration, characterized in that, The method includes: Obtain the unlabeled dataset for the target language; Pseudo-labels are generated for each sample in the target language dataset using a pre-trained source language model and an initialized target language model. Determine the reliability weight of the pseudo-label for each sample in the unlabeled dataset of the target language, and construct a weighted cross-entropy loss function based on the reliability weight; The parameters of the target language model are iteratively updated based on the weighted cross-entropy loss until the performance indicators of the target language model on the target language validation set meet the preset conditions or reach the maximum number of iterations. Then the iteration stops, and the optimized target language model is obtained. The text to be classified is input into the optimized target language model for analysis and processing to obtain the classification result.

2. The method according to claim 1, characterized in that, The determination of the reliability weight of the pseudo-label for each sample in the unlabeled dataset of the target language includes: The pre-trained language model encoder is used to encode each word in the unlabeled dataset of the target language to generate a word representation vector, and the representation vectors of all words are stored in the memory bank. Nearest neighbor retrieval is performed in the memory based on the cosine similarity between the representation vectors to determine the nearest neighbor instances of each word; Calculate the consistency score between each word pseudo-label and its nearest neighbor instance pseudo-labels; The consistency score is converted into a reliability weight corresponding to each word pseudo-label.

3. The method according to claim 2, characterized in that, The cosine similarity satisfies the following formula: ； in, For words The representation vector, For words The representation vector, for The nearest neighbor instance.

4. The method according to claim 2, characterized in that, The consistency score between each word pseudo-tag and its nearest neighbor instance pseudo-tags is calculated according to the following formula: ； in, For the source language model, words Predicted soft tags, For words Nearest neighbor instances, For words The number of nearest neighbor instances, For indicator functions, As a hard label, As a hard label, For the source language model, words Predicted soft labels.

5. The method according to claim 4, characterized in that, The process of converting the consistency score into a reliability weight corresponding to each word pseudo-label satisfies the following formula: ； in, For hyperparameters, It refers to all predictions as categories in the unlabeled dataset of the target language. The average consistency score of the words.

6. The method according to claim 1, characterized in that, The weighted cross-entropy loss function satisfies the following formula: ； in, It is the sequence length. It is the cross-entropy loss function.

7. The method according to claim 1, characterized in that, The method further includes: The optimized target language model is used to generate pseudo-labels for each sample in the unlabeled target language dataset. A consistency check is performed between the pseudo-labels generated by the optimized target language model and the pseudo-labels generated by the pre-trained source language model, and samples with inconsistent pseudo-labels are re-labeled.

Images