Multilingual text classification method, system and device

Abstract

The present invention discloses a multilingual text classification method, system, and device. The method comprises: obtaining a multilingual text dataset; constructing a text classification model, wherein the text classification model comprises a text representation module, a language fusion module, a text debiasing module, and a text classification module; training the text classification model based on the multilingual text dataset to obtain a trained text classification model; and classifying a test text based on the trained text classification model. The system comprises: a data acquisition unit, a model construction unit, a model training unit, and a classification module. The device comprises a memory and a processor for executing the multilingual text classification method. By using the present invention, the problem that existing multilingual text classification tasks lack sufficient fairness can be solved. The present invention can be widely applied in the field of data classification.

Description

Technical Field

[0001] The present invention relates to the field of data classification, and in particular to a multilingual text classification method, system and device. Background Art

[0002] Because fairness is a complex and abstract issue, different fields use different fairness metrics for different tasks, and each metric measures different degrees of fairness. Text classification tasks involve fairness assessment and bias removal. While many methods exist for assessing and mitigating bias, most are conducted in resource-rich monolingual environments. Research on fair classification of multilingual text is insufficient.

[0003] However, multilingual text classification is more complex than monolingual text classification. Therefore, there is an urgent need for a method that can be applied to multilingual text classification tasks. Summary of the Invention

[0004] In view of this, in order to solve the problem that existing multilingual text classification tasks are not sufficiently fair, the present invention proposes a multilingual text classification method, which includes the following steps:

[0005] Obtain multilingual text datasets;

[0006] Constructing a text classification model, wherein the text classification model includes a text representation module, a language fusion module, a text debiasing module, and a text classification module;

[0007] Train the text classification model based on the multilingual text dataset to obtain a trained text classification model;

[0008] Classify the test text based on the trained text classification model.

[0009] In this step, the text classification model is used as a debiasing framework.

[0010] In some embodiments, further comprising:

[0011] Construct a fairness evaluation framework for multilingual text classification and evaluate the classification results.

[0012] Through this optimization step, a fairness evaluation framework applicable to multiple languages ​​is constructed.

[0013] In some embodiments, the text classification model, wherein:

[0014] Text representation module, used to encode and represent text;

[0015] The language fusion module narrows the semantic distance between samples with the same target label in different languages ​​through contrastive learning;

[0016] The text debiasing module shortens the semantic distance between samples with the same target label but different sensitive attribute values ​​through contrastive learning;

[0017] A text classification module for classification based on phonetic distance and text representation.

[0018] In some embodiments, the step of training the text classification model based on the multilingual text dataset to obtain a trained text classification model specifically includes:

[0019] Train text classification models based on multilingual text datasets;

[0020] Construct a contrastive loss function as the training target for the language fusion module and the text debiasing module;

[0021] Construct a cross entropy loss function as the training objective for the text classification module;

[0022] Back propagation updates the model parameters to obtain the trained text classification model.

[0023] In this preferred step, loss functions are constructed for different modules as training targets, which makes it easier to complete model fitting.

[0024] In some embodiments, the contrastive loss function for each cross-batch sample i in the language fusion module is:

[0025]

[0026] Among them, sim(·) represents the cosine similarity function, v i represents the semantic representation of the sample with subscript i, v t Represents the semantic representation of the sample with subscript t, T represents the set of all other samples in the batch that share the same target label with sample i, t represents the index of a sample in T, and k represents an index in the set of index set I after removing the subscript i.

[0027] First, the loss of each cross-batch is calculated, and then the loss of the language fusion module for the entire batch is calculated.

[0028] In some embodiments, the cross entropy loss function of the text classification module is:

[0029]

[0030] In the above formula, Y represents the category set of the text classification task, represents the one-hot value of the nth label, P n It is expressed as the predicted probability of the nth expression.

[0031] Subsequently, weights can be set separately according to the loss function mentioned above and combined to generate the overall loss.

[0032] In some embodiments, the multilingual text classification fairness assessment framework specifically includes:

[0033] Construct different evaluation indicators to calculate the monolingual equality difference, multilingual equality difference and multilingual equality performance difference;

[0034] Defines how destructive the debiasing operation is to other sensitive attributes.

[0035] Through this optimization step, the fairness of single language and multilingual is calculated respectively, and the fairness of the debiasing operation is also defined and evaluated.

[0036] The present invention also proposes a multilingual text classification system, comprising:

[0037] A data acquisition unit, used to acquire a multilingual text dataset;

[0038] A model building unit, configured to build a text classification model, wherein the text classification model includes a text representation module, a language fusion module, a text debiasing module, and a text classification module;

[0039] Model training unit, which trains the text classification model based on a multilingual text dataset to obtain a trained text classification model;

[0040] The classification unit classifies the test text based on the trained text classification model.

[0041] The present invention also proposes a multilingual text classification device, comprising:

[0042] at least one processor;

[0043] at least one memory for storing at least one program;

[0044] When the at least one program is executed by the at least one processor, the at least one processor implements the multilingual text classification method as described above.

[0045] Based on the above scheme, the present invention provides a multilingual text classification device, system and device, and builds a debiasing framework for multilingual text classification. The text representation module uses a multilingual pre-trained language model to represent text, so that data in different languages ​​can be represented using the same encoding method. By using contrastive learning, the language fusion module reduces the distance between samples with the same target label in different languages. Therefore, the semantic spaces of different languages ​​tend to be consistent, and training data in multiple languages ​​continuously improves the overall performance of the model. The text debiasing module adopts the concept of contrastive learning to narrow the distance between samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the steps of a multilingual text classification method of the present invention;

[0047] Figure 2 It is a structural block diagram of a multilingual text classification system of the present invention. DETAILED DESCRIPTION

[0048] Text classification automatically categorizes and labels text collections according to a specific classification system or standard. Based on a set of labeled training documents, it finds a relationship model between document features and document categories. It then uses this learned relationship model to classify new documents. Text classification has gradually evolved from knowledge-based approaches to those based on statistics and machine learning.

[0049] Contrastive learning is an unsupervised learning method that aims to learn useful feature representations by comparing data samples with other samples. It does this by bringing similar samples closer together in feature space and separating dissimilar samples, thereby enabling the model to learn the underlying structure of the data. The core idea of ​​contrastive learning is to maximize the similarity between similar pairs of samples and minimize the similarity between dissimilar pairs of samples.

[0050] In response to the problem raised in the background technology that existing multilingual text classification tasks are not sufficiently fair, with the development of machine learning / deep learning, the present invention combines contrastive learning technology to reduce the distance between samples with the same target label in different languages ​​to achieve fair classification of multilingual texts.

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] It should be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other.

[0053] It should be understood that the terms "system," "device," "unit," and / or "module" used in this application are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0054] As used in this application and the claims, unless the context clearly indicates an exception, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular and may include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements. The phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus that includes the elements.

[0055] In the description of the embodiments of this application, "plurality" refers to two or more than two. The terms "first" and "second" below are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0056] In addition, flow charts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0057] Reference Figure 1 , which is a flow chart of an optional example of the multilingual text classification method proposed in the present invention. The method can be applied to a computer device. The imaging method proposed in this embodiment may include but is not limited to the following steps:

[0058] Step S1: Obtain a multilingual text dataset;

[0059] Given a sample set G = {(W1, y1, s1, l1), (W2, y2, s2, l2), ..., (W K ,y K , s K , l K )}, I = {1, ..., K} is the index of the sample, where W i ={w1, w2, ..., w n}(1≤i≤K) is the text sequence of sample i, and n is the sentence length of sample i. i is the target label of sample i, where y i ∈Y. Y is the category set of the text classification task. i is the sensitive attribute that needs to be debiased, where s i ∈S, S is a set of sensitive attribute values. Taking gender as an example, the set S contains two sensitive attribute values, namely male and female.i is the language of sample i, where l i ∈L, where L is all the languages ​​included in the multilingual text classification task. It is worth noting that under the multilingual text task definition, during the training process, the model will receive sensitive attributes and language attributes for training, but during the testing process, the model will not use sensitive attributes and language attributes as known information.

[0060] Step S2: constructing a text classification model;

[0061] The text classification model includes a text representation module, a language fusion module, a text debiasing module and a text classification module;

[0062] Text representation module:

[0063] The text representation module aims to represent texts in different languages ​​using the same encoding method, so that the semantic representations of multiple languages ​​are distributed in the same semantic space. We use the mBERT (Multilingual Bidirectional Encoder Representation from Transformer) pre-trained model to encode and represent the text. The BERT model is a pre-trained model based on Transformer. By pre-training with the NSP task and the MLM task, the model has rich semantic representations. After using word fragments to segment the text, the word units, segmentation, and position embeddings of the text are extracted. In addition, in general, we use a special token [CLS] as its sentence vector representation. Therefore, our multilingual representation module uses the mBERT model as the base model to encode language features, and uses the first input token [CLS] to obtain the sentence vector expression. Therefore, for the i-th sentence W i , its sentence vector is represented as follows:

[0064] v i =mBERT(a i , b i , c i )

[0065] where a i , b i , c i They are word embedding, segment embedding and position embedding respectively.

[0066] Language Fusion Module:

[0067] The language fusion module aims to make the representation of different languages ​​in the semantic space more compact. In the language fusion module, contrastive learning is used to narrow the distance between samples with the same target label in different languages, so that the semantic spaces of different languages ​​tend to be consistent.

[0068] Text debiasing module:

[0069] Unlike other debiasing methods that rely on external resources such as sensitive attribute dictionaries, the text debiasing module achieves a more general debiasing operation through contrastive learning. While the language fusion module shortens the semantic distance between samples with the same label but different languages, the text debiasing module shortens the semantic distance between samples with different sensitive attribute values ​​for the same label. The text debiasing module aims to make the representations of samples with different sensitive attribute values ​​closer in the semantic space, thereby confusing the model when identifying sensitive attributes. In other words, we want samples with different sensitive attribute values ​​to be indistinguishable.

[0070] Text Classification Module:

[0071] In the text classification module, we take the output of the multilingual text representation module as input to the feedforward layer with K×H dimensional weights, denoted as W, where K is the number of labels. We denote this intermediate representation as F with a dimension of 1×K.

[0072] F=v i W T +b

[0073] where · represents the weight matrix W and the context vector v i The final label probability is calculated by the standard softmax function through the intermediate representation. The output P of the softmax layer has a dimension of 1×K.

[0074] P = softmax(F)

[0075] Step S3: training the text classification model based on the multilingual text dataset to obtain a trained text classification model;

[0076] Step S4: classify the text to be tested based on the trained text classification model.

[0077] In some embodiments, the step of training the text classification model based on the multilingual text dataset to obtain a trained text classification model specifically includes:

[0078] Train text classification models based on multilingual text datasets;

[0079] Construct a contrastive loss function as the training target for the language fusion module and the text debiasing module;

[0080] In the language fusion module:

[0081] For a given sample i, all other samples in the batch that share the same target label constitute the set T = {t: t∈I, y t =y i ∧l t ≠l i∧t≠i}. Among them, the sample set T contains samples of different languages. Then we can define the contrastive loss function of the language fusion module for each cross-batch entry i as:

[0082]

[0083] Among them, sim(·) represents the cosine similarity function, v i represents the semantic representation of the sample with subscript i, v t Represents the semantic representation of the sample with subscript t, T represents the set of all other samples in the batch that share the same target label with sample i, t represents the index of a sample in T, and k represents an index in the index set / the set after removing the subscript i.

[0084] The loss of the language fusion module for the entire batch is:

[0085]

[0086] Where |G| represents the number of samples in a batch.

[0087] In the text debiasing module:

[0088] For a given sample i, all other samples that share different sensitive attributes with it in the batch constitute a set Q = {q:q∈I,y q =y i ∧S q ≠S i ∧q≠i}. Then define the contrast loss function of the text debiasing module for each entry i across batches as:

[0089]

[0090] For the entire batch, the loss value obtained by the text debiasing module is:

[0091]

[0092] Where |G| represents the number of samples in a batch.

[0093] Construct a cross entropy loss function as the training objective for the text classification module;

[0094] The text classification module uses cross entropy loss as the training target:

[0095]

[0096] In the above formula, Y represents the category set of the text classification task, represents the one-hot value of the nth label, P n It is expressed as the predicted probability of the nth expression.

[0097] Back propagation updates the model parameters to obtain the trained text classification model.

[0098] The overall training loss of the framework is the overall loss of the language fusion module, text debiasing module, and text classification module:

[0099] L=α·L li +β·L td +(1-α-β)·L ce

[0100] Among them, α and β are loss weights, which are used to balance the learning intensity of each module.

[0101] In some embodiments, the method further includes constructing a fairness evaluation framework for multilingual text classification to evaluate the classification results. The fairness evaluation framework for multilingual text classification includes:

[0102] 1. Monolingual equality differences

[0103] Similar to other existing multilingual fairness studies, we measure the monolingual equality difference (MED) for multilingual models.

[0104] We use the classic monolingual text classification fairness evaluation metric, FPED, as a measure of monolingual equality difference. FPED combines the difference between the false positive rate (FPR) within a specific sensitive attribute group and the false positive rate of all data. We calculate this metric using the following formula:

[0105]

[0106] 2. Multilingual equality differences

[0107] Unlike single-language fairness evaluation, multilingual fairness focuses on evaluating the fairness of the model across all languages. That is, when calculating FPR, the statistical result is obtained by combining the test set of all languages, namely:

[0108]

[0109] Among them, FPR(L, d) is the false positive rate calculated for the test set samples with label d for sensitive attribute D in all languages, FPR L The false positive rate (MUED) is calculated for all test set samples across all languages. In the field of multilingual natural language processing (NLP) models, the MUED value is crucial as an indicator of the model's multilingual bias. The MUED value is positively correlated with the degree of multilingual bias evident in the model; larger values ​​indicate greater multilingual bias. Lower MUED values ​​indicate increased fairness across different languages.

[0110] 3. Differences in Multilingual Equality

[0111] Multilingual group fairness requires not only that multilingual models be fair on sensitive attributes, but also that they achieve equal predictive performance across different languages. From a representational perspective, this requirement is difficult to achieve without clearly defined tasks and metrics. We represent equal performance differences across multiple languages ​​by balancing the macro-average F-score across languages. Given a text classification model, its macro-average F-score on the test set of different languages ​​is:

[0112]

[0113] Next, we define equal performance differences across multiple languages ​​by averaging the accuracy across languages, treating languages ​​as group members. If for all languages ​​l i ,l j , Then the multilingual model M satisfies the multilingual F-score parity. In practice, it is impossible to achieve equal F-scores for all languages. We first calculate the mean of the macroaverage F-scores for all languages:

[0114]

[0115] Then, similar to how we calculate fairness for a single language, we sum the difference between the performance of a specific language and the performance of all languages ​​and calculate it by the following formula:

[0116]

[0117] The MEPD value plays a crucial role in measuring the degree of performance imbalance exhibited by a model across different languages. There is a positive correlation between the MEPD value and the degree of performance imbalance: higher MEPD values ​​indicate greater performance imbalance. Conversely, a negative correlation is observed: lower MEPD values ​​indicate a model's ability to achieve cross-lingual performance consistency in a multilingual performance model.

[0118] 4. Negative returns on strategies

[0119] When a model debiass a particular sensitive attribute, it may inadvertently amplify bias in other sensitive attributes, which is an undesirable outcome. Therefore, we further evaluate whether the model increases bias in one sensitive attribute while debiasing another. We want the model's debiasing to focus not only on the specific sensitive attribute to be debiased, but across all sensitive attributes. We believe that when a model debiass one sensitive attribute, it should not negatively impact other sensitive attributes. For example, consider a model debiasing the sensitive attribute of gender. After debiasing the model, we need to evaluate other sensitive attributes such as race and age. If a model debiasss gender but increases bias in other attributes such as race or age, it should not be considered "fair."

[0120] Given that the current sensitive attribute to be debiased is S, and the other existing sensitive attributes are O={O1,…,O z}, assuming that the fairness results of the model without debiasing operation under other sensitive attributes O are MED O The model fairness result obtained by debiasing attribute S under other sensitive attributes O is The destructiveness of the debiasing operation on other sensitive attributes is defined as:

[0121]

[0122] This paper takes a novel perspective, focusing not only on the model's performance on multilingual fairness tasks but also on evaluating the fairness of multilingual classification models from multiple dimensions. It proposes a multi-dimensional fairness assessment framework for multilingual text classification, evaluating the model's fairness across individual languages, across all languages, across multiple languages, and the destructiveness of fairness strategies.

[0123] In terms of the selection of comparison methods, we only selected the following five methods:

[0124] BERT (Individual Training): As our debiasing method is applied to BERT, we fine-tune the BERT model to measure performance and fairness on the hate speech identification task. We use the dataset for each language to train a BERT-based hate speech classification model separately, which we call "individual training."

[0125] BERT (Merged Training): Unlike the separate training mode, the merged training mode utilizes all languages ​​in the multilingual dataset and only trains a single hate speech classification model.

[0126] BERT+FGM: FGM introduces an innovative approach to text by extending adversarial and virtual adversarial training techniques. Unlike previous methods that perturb the raw input, FGM perturbs the word embeddings in the neural network. Through its implementation, FGM achieves outstanding performance on various benchmark tasks in both semi-supervised and purely supervised settings.

[0127] BERT+PGD: PGD is an iterative attack. Unlike FGM, it performs multiple iterations, taking smaller steps each time and projecting the perturbations within a specified range. PGD optimizes and discovers adversarial examples, which are then used for training, resulting in a neural network that minimizes the loss against these adversarial examples. This makes the neural network resilient to further adversarial examples. Furthermore, PGD successfully addresses the challenges posed by internal non-convex optimization.

[0128] SENT-DEBIAS BERT: SENTDEBIAS represents a significant advancement in debiased sentence representation technology, especially in terms of generalization. SENT-DEBIAS effectively contextualizes biased attribute words by covering the diverse uses of biased attribute words in natural sentences by integrating different sentence templates from various text corpora. This process produces biased attribute sentences that successfully mitigate the biases associated with binary and multi-class attributes. A key component of SENTDEBIAS is its contextualization stage, where biased attribute words are converted into biased attribute sentences by using a large number of sentence templates from text corpora.

[0129] The main experimental results are as follows:

[0130] As shown in Tables 1-4, we conducted experiments on four sensitive attributes: age, country, class, and gender, and used five evaluation metrics: accuracy, macro-average F-value, micro-evaluation F-value, AUC, and MED.

[0131] Table 1: Experimental results of different models on age attribute debiasing

[0132]

[0133] Table 2: Experimental results of different models on gender attribute debiasing

[0134]

[0135]

[0136] Table 3: Experimental results of different models on class attribute debiasing

[0137]

[0138]

[0139] Table 4: Experimental results of different models on national attribute debiasing

[0140]

[0141]

[0142] like Figure 2 As shown, a multilingual text classification system includes:

[0143] A data acquisition unit, used to acquire a multilingual text dataset;

[0144] A model building unit, configured to build a text classification model, wherein the text classification model includes a text representation module, a language fusion module, a text debiasing module, and a text classification module;

[0145] Model training unit, which trains the text classification model based on a multilingual text dataset to obtain a trained text classification model;

[0146] The classification unit classifies the test text based on the trained text classification model.

[0147] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] A multilingual text classification device:

[0149] at least one processor;

[0150] at least one memory for storing at least one program;

[0151] When the at least one program is executed by the at least one processor, the at least one processor implements the multilingual text classification method as described above.

[0152] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] A storage medium stores processor-executable instructions, which, when executed by a processor, are used to implement the multilingual text classification method described above.

[0154] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0155] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A multilingual text classification method, characterized in that: The following steps are involved: Obtain multilingual text datasets; Constructing a text classification model, wherein the text classification model includes a text representation module, a language fusion module, a text debiasing module, and a text classification module; Train the text classification model based on the multilingual text dataset to obtain a trained text classification model; Classify the test text based on the trained text classification model; Construct a fairness evaluation framework for multilingual text classification and evaluate the classification results; The text classification model, wherein: Text representation module, used to encode and represent text; The language fusion module narrows the semantic distance between samples with the same target label in different languages ​​through contrastive learning; The text debiasing module shortens the semantic distance between samples with the same target label but different sensitive attribute values ​​through contrastive learning; Text classification module, used for classification based on phonetic distance and text representation; The step of training the text classification model based on the multilingual text dataset to obtain a trained text classification model specifically includes: Train text classification models based on multilingual text datasets; Construct a contrastive loss function as the training target for the language fusion module and the text debiasing module; Construct a cross entropy loss function as the training objective for the text classification module; Back propagation updates the model parameters to obtain the trained text classification model; The contrast loss function for each cross-batch sample i in the language fusion module is: In the above formula, sim(·) represents the cosine similarity function, v i represents the semantic representation of the sample with subscript i, v t Represents the semantic representation of the sample with subscript t, T represents the set of all other samples in the batch that share the same target label with sample i, t represents the index of a sample in T, and k represents an index in the set of index set I after removing the subscript i.

2. A multilingual text classification method according to claim 1, characterized in that: The cross entropy loss function of the text classification module is: In the above formula, Y represents the category set of the text classification task, represents the one-hot value of the nth label, P n It is expressed as the predicted probability of the nth expression.

3. A multilingual text classification method according to claim 1, characterized in that: The multilingual text classification fairness evaluation framework specifically includes: Construct different evaluation indicators to calculate the monolingual equality difference, multilingual equality difference and multilingual equality performance difference; Defines how destructive the debiasing operation is to other sensitive attributes.

4. A multilingual text classification system, characterized in that: The method for performing the multilingual text classification method according to claim 1 comprises: A data acquisition unit, used to acquire a multilingual text dataset; A model building unit, configured to build a text classification model, wherein the text classification model includes a text representation module, a language fusion module, a text debiasing module, and a text classification module; Model training unit, which trains the text classification model based on a multilingual text dataset to obtain a trained text classification model; The classification unit classifies the test text based on the trained text classification model.

5. A multilingual text classification device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the multilingual text classification method according to any one of claims 1 to 3.

Images