Fair representation learning method for text data
Through non-gradual rate distortion measurement functions and embedded space normalization technology, a fair representation space is built, which solves the complex challenges brought by high-dimensional data and unknown distribution, and realizes the information integrity and accuracy of text data in embedded space, improving the The fairness and stability of the model.
Patent Information
- Application Number
- CN202410885358.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-12-03
AI Technical Summary
When building fair representation spaces, it is difficult to effectively process high-dimensional data and unknown distributions, resulting in poor construction of fair representation spaces, and there are difficulties in optimizing and training instability in adversarial network methods.
Non-progressive rate distortion measurement function and embedding space normalization technology are used to convert text data into a word list embedding matrix, embedding representation vectors are generated using the BERT network, loss functions are designed to meet fairness constraints, and fair representation spatial model is constructed.
Ensure that data maintains the integrity and accuracy of original information when converted to embedded space, improve the generalization ability and robustness of the model, avoid unfair phenomena caused by data bias, and enhance the trust and acceptance of the model.
Smart Images

Figure CN119067166A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of representation learning, and in particular to a fair representation learning method for text data. Background Art
[0002] Representation space construction: refers to the process of converting raw data into a high-dimensional feature space in the field of machine learning. Through representation space construction, the complex structure of raw data can be converted into a more compact and manageable representation, making it easier for machine learning algorithms to process and learn.
[0003] Fair representation refers to the processing and transformation of data or information to generate data representations or information representations that can maintain high quality and be credible at all stages. Fair representation learning aims to eliminate noise, redundancy, and attribute bias in data, thereby extracting important information that can effectively describe the nature and characteristics of the data.
[0004] As a core branch in the field of machine learning, representation learning has the fundamental idea of revealing the internal structure of data and the relationship between features more deeply by learning the representation or features of data. Its main purpose is to discover the effective representation of data through automated learning methods, and on this basis simplify, optimize and improve the accuracy of subsequent learning tasks. However, with the development of technology and the increase in demand, in order to build a more credible representation space, fair representation methods have emerged. Fair representation methods can map raw data to a more representative and robust feature space through a series of sophisticated data preprocessing and feature extraction processes. This conversion not only enables the machine learning system to output more credible and reliable results, but also largely ensures the stability and fairness of the system in practical applications. The present invention will elaborate on key technologies and theories such as fair representation learning methods based on metric functions, and list the experimental schemes that are most similar to the present invention.
[0005] Fair representation learning aims to find a feature transformation method to process and transform data or information to generate data representation or information representation that can maintain high quality, credibility and reliability in all links. By eliminating noise, redundancy and uncertainty in the data, fair representation learning can extract important information that effectively describes the nature and characteristics of the data.
[0006] In order to better understand fair representation learning, the commonly used fairness definitions and general optimization forms are introduced in detail below:
[0007] In the real world, different machine learning tasks have different focuses, and it is difficult to determine a universal fairness definition. This paper focuses on statistical fairness. It requires that the treatment of protected groups is similar to that of non-vulnerable groups or the entire group, without making additional assumptions about the data and is easy to verify. For the convenience of description, it is assumed that the protected attribute S is a binary attribute S∈{0,1}. Common fairness definitions are:
[0008] (a) Demographic parity (DP)
[0009] If the prediction value of the machine learning model In any case, it is independent of the protected attribute S, and the model is considered to satisfy population parity (DP), that is:
[0010]
[0011] This definition requires that different groups obtain the same output prediction results with the same probability, which is usually approximated by the following two approximate forms:
[0012]
[0013] Among them, ε1 and ε2 both represent approximate control parameters, and this approximate form can be supported by the "four-fifths rule" in law. However, if the target variable Y is highly correlated with the protected attribute S, the utility of this definition will be limited. For example, in candidate screening, selecting qualified members from one group and randomly selecting members from another group with the same probability, although this lazy practice meets the definition of population equality, loses its practical significance.
[0014] (b) Equality of opportunity (EO)
[0015] Equal opportunity measures the difference between the error rate and accuracy of the output results of each group. If the probability of predicting labels based on different protected attributes S in the same category Y is equal, then the model is considered to meet equal opportunity (EO), that is,
[0016]
[0017] General optimization form
[0018] Typically, the goal of fair representation learning is to minimize the loss function L(D; θ) while satisfying the fairness constraint R(D; θ), which is generally expressed as:
[0019]
[0020] in, represents the training data containing N samples, x represents the data features used in model training, y and s represent the corresponding data labels and protected attribute values respectively. represents the loss of the target task, such as classification or regression, and the second is a designed fairness regularizer. The parameter λ is used to weigh the trade-off between fairness and utility. Common target task loss functions include cross entropy loss and contrast loss, which are introduced below:
[0021] Cross Entropy is an important concept in Shannon information theory, which is mainly used to measure the difference between two probability distributions. In information theory, cross entropy represents two probability distributions p and q, where p represents the true distribution and q represents the untrue distribution. In the same set of events, the untrue distribution q is used to represent the average number of bits required for an event to occur. Let's take an example to describe it:
[0022] Suppose there are two probability distributions p and q in a sample set, where p is the true distribution and q is the false distribution. If the expected encoding length required to identify a sample is measured according to the true distribution p, then:
[0023]
[0024] However, if the wrong distribution q is used to represent the average encoding length of the true distribution p, it should be:
[0025]
[0026] At this time, H(p,q) is called cross entropy, which is calculated as follows:
[0027] Discrete variables:
[0028] Continuous variable: –∫P(x)log Q(x)dr(x)=E p [―logQ]
[0029] Contrastive loss is an unsupervised loss function for learning data representation, which aims to maximize the consistency between similar samples and minimize the consistency between dissimilar samples. The core idea is to cluster similar samples together in the representation space through learning, while dissimilar samples are dispersed. The general form of the loss function is as follows:
[0030]
[0031] The first term of the contrast loss measures the positive sample pair (x i , xj ) in the representation space, hoping that their representations are as close as possible; while the second term is obtained by negative samples (x i ,x j ) is used to measure the similarity between the representations of dissimilar sample pairs, hoping that their representations are as dissimilar as possible. By maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the contrastive loss expects the model to learn a representation space that can effectively capture the intrinsic structure and characteristics of the data, thereby better distinguishing similar samples from dissimilar samples.
[0032] The core idea of regularizer design is to quantify and minimize the correlation between the protected attribute and the prediction. Commonly used metric functions are MMD distance, mutual information and KL divergence, which are introduced below:
[0033] Maximum Mean Discrepancy distance (MMD) is a statistical method used to measure the difference between two probability distributions. Its basic idea is to measure the distance between them by comparing the difference in the mean function of the sample feature space of the two distributions. If the two distributions are similar, then their mean functions in the feature space should be similar, otherwise the difference is large. Specifically, given two probability distributions P and Q, and their sample set X = {x1, x2, ..., x n}and The calculation formula of MMD distance is as follows:
[0034]
[0035] Where f(·) is a mapping function that maps the input to a feature space (usually a Hilbert space), H is the kernel Hilbert space of the feature space, and ||·|| H is the norm in this space.
[0036] Mutual Information is a quantitative indicator used in information theory to measure the degree of mutual dependence between two random variables. It indicates the degree to which the amount of information in a random variable can be used to describe the uncertainty of another random variable. The calculation of mutual information is based on the joint distribution of two random variables and their respective marginal distributions. Assuming that X and Y are two random variables, their joint distribution is p(x,y), and their marginal distributions are p(x) and p(y), respectively, then the mutual information I(X;Y) between them can be calculated by the following formula:
[0037]
[0038] The calculation of mutual information includes the comparison of joint probability distribution and marginal probability distribution to measure the correlation between two random variables. If two random variables are independent, their mutual information is 0; if there is a correlation between them, the mutual information is greater than 0. The larger the mutual information, the stronger the correlation between the two random variables.
[0039] KL (Kullback-Leibler) divergence, also known as relative entropy, is a measure used in information theory to measure the difference between two probability distributions. It quantifies the "distance" between two probability distributions and is used to measure the information loss when using distribution Q to approximate the true distribution P. The specific calculation formula is as follows:
[0040]
[0041] Among them, P(x) and Q(x) are the probabilities of probability distributions P and Q at value x respectively.
[0042] At present, one of the existing methods for constructing a fairness representation space includes: FCMI (Fair Clustering via Mutual Information) uses mutual information as a correlation measurement function to constrain the discriminability and fairness of representation in the embedding space. It follows the original definition of fair representation and uses mutual information to measure the correlation between the learned representation Z and the original data X, the target variable Y and the protected attribute S respectively.
[0043] The optimization goal of this method is:
[0044] max I(Z;Y|S)-β(I(Z;X)-I(Z;X|S))
[0045] Since mutual information is difficult to calculate directly in high-dimensional continuous space, the algorithm further transforms the above formula:
[0046] I(Z; Y|S)≥--CE(Z; Y)
[0047] I(Z;X|S)≥E x,z,s log q(x|z,S)
[0048] I(Z;X)≤E x KL(q(z|x)|p(z))
[0049] The general process of the algorithm is:
[0050] ① Initialization parameter θ
[0051] ②For the input sample x, calculate the embedded representation z = f(x; θ)
[0052] ③ Calculate the mutual information I(Z;Y|S) between the embedded representation z and the corresponding label y given the protected attribute S
[0053] ④ Calculate the mutual information I(Z;X) between the embedded representation z and the original data x
[0054] ⑤ Calculate the mutual information I(Z;X|S) between the embedded representation z and the original data x given the protected attribute S
[0055] ⑥ Calculate the loss function gradient Δθ and update the network parameters:
[0056] θ t+1 =θ t +α·Δθ
[0057] If the number of iterations is reached, the training is completed, otherwise go to ②
[0058] The disadvantages of the above-mentioned method for constructing a fairness representation space in the prior art include: the method uses variational inference and discrete sampling to approximate mutual information, and is therefore easily affected by sampling frequency and distribution estimation.
[0059] Prior art metric function-based methods attempt to approximate the correlation between variables through variational inference or discrete sampling, but this approach often does not work well with high-dimensional data and unknown distributions. This is because the complex relationships in high-dimensional data are difficult to accurately capture with simple metric functions, and unknown distributions make the variational inference and sampling processes more difficult. On the other hand, adversarial network-based methods, while attractive in theory, face optimization difficulties and fairness issues in practical applications. Adversarial networks simulate potential sources of discrimination by introducing an "attacker" and train the model to defend against this attack. However, this approach often fails to achieve true fairness because the performance of the attacker limits the performance of the model. In addition, adversarial networks themselves also face problems of unstable training and difficulty in convergence. Summary of the invention
[0060] The present invention provides a fair representation learning method for text data to ensure that the integrity and accuracy of the original information of the data can be maintained when it is converted to an embedding space.
[0061] In order to achieve the above object, the present invention adopts the following technical scheme.
[0062] A fair representation learning method for text data, characterized by comprising:
[0063] Convert the collected text data into a vocabulary, and use the vocabulary to construct a vocabulary embedding matrix consisting of discrete vectors;
[0064] Using BERT to convert the vocabulary embedding matrix into an embedding representation vector;
[0065] According to the influence of the protected attributes and cluster structure on the information encoding of the original text data measured by the embedded representation vector, a loss function is designed based on the measurement result of the non-progressive distortion measurement function, a deep neural network is trained using the loss function, and a fair representation space model is constructed based on the trained deep neural network; the text data to be processed is cleaned and input into the fair representation space model, and the fair representation space model outputs a fair representation of the text data to be processed.
[0066] Preferably, the step of converting the collected text data into a vocabulary and constructing a vocabulary embedding matrix composed of discrete vectors using the vocabulary comprises:
[0067] The collected online biographical text data are preprocessed, including word segmentation, removal of stop words and punctuation marks, and lemma restoration or stemming. A vocabulary is constructed based on the preprocessed text, and the vocabulary is used to map each word in the text data into a discrete vector. Each vector has the same dimension as the number of words in the vocabulary, and only one position has a value of 1, and the values of the remaining positions are 0. The position with a value of 1 corresponds to the index of the word in the vocabulary, thereby converting the text data into a vocabulary embedding matrix composed of discrete vectors, in which each row represents a sentence and each column represents a word in the vocabulary.
[0068] Preferably, the converting the vocabulary embedding matrix into an embedding representation vector using BERT comprises:
[0069] The bidirectional encoder representation technology BERT network is used to map the sentences in the vocabulary embedding matrix to the embedding space to obtain the embedding representation vector of the sentence. The embedding representation vector of the sentence of the BERT network is expressed by the following formula:
[0070] Z=F([CLS]+X)
[0071] Wherein, Z is the embedding representation vector of the sentence, F(~) is the pre-trained deep neural network model, [CLS] is a marking symbol used to represent the overall semantics of the sentence, + represents the concatenation operation of the vector, and X is the sentence in the embedding matrix of the vocabulary;
[0072] By first concatenating the input sentence with a special marker symbol [CLS] and then inputting it into the pre-trained deep neural network model, the model output is the sentence embedding representation vector Z = F(X; Θ). This embedding representation vector Z retains the semantic and contextual information of the sentence.
[0073] Preferably, the measuring the influence of the protected attribute and the cluster structure on the encoding of the original text data information according to the embedded representation vector includes:
[0074] The transformed embedding representation vector Z is evaluated using a non-progressive rate-distortion metric function. Without loss of generality, given a dataset Where x represents the data features used in model training, y and s represent the corresponding data labels and protected attribute values, respectively. ε is set as the encoding error, and the average number of bits required to re-encode the input data is estimated:
[0075]
[0076] in, is the embedded representation after data transformation, D is the embedding representation dimension, and N is the number of samples.
[0077] According to the cluster structure, the embedding representation vector Z is divided into multiple data subsets Z = Z1∪Z2∪...∪Z K , each subset is located in a low-dimensional subspace. For each data subset, the average number of codes required to encode the subset samples is measured:
[0078]
[0079] in, is a set of diagonal matrices, whose diagonal elements represent the membership relationship between N samples and K clusters, that is, Π i (j, j)∈[0, 1] represents the probability that sample j belongs to cluster i.
[0080] According to the protected attribute value of the sample, the average number of codes required to encode samples with different protected attributes in the same data subset Zi is estimated:
[0081]
[0082] Among them, Z i is the data subset of the i-th class, N i is the number of samples contained in the i-th data subset, is the diagonal matrix of membership of different protected attributes corresponding to the data subset, Indicates whether sample j contains the protected attribute m.
[0083] The non-progressive rate characteristic of the non-progressive distortion metric function can be used to evaluate the information coding quality within a given ε range.
[0084] Preferably, the designing of a loss function based on the measurement result of the non-progressive distortion measurement function, using the loss function to train a deep neural network, and constructing a fair representation space model based on the trained deep neural network include:
[0085] A data transformation function F(X; Θ) is obtained, which can transform the original data into a more representative and fair feature space, ensuring that the obtained embedded representation vector satisfies the fairness constraint, which is defined as follows:
[0086] R(Z,ε)-R(Z,ε|Π s )=0
[0087] Among them, R(Z, ε) and R(Z, ε|Π s ) respectively measure the average number of codes required to encode the entire text data of the input and samples with different protected attributes, Z = F(X; Θ) represents the obtained embedding representation, ε is the allowed encoding error, is the corresponding protected attribute membership diagonal matrix, represents whether sample j contains the protected attribute m. The fairness constraint requires that the protected attribute does not affect the encoding rate of the data. If the fairness constraint can be met, the obtained embedding representation vector Z is considered fair;
[0088] In order to measure the impact of protected attributes and cluster structures on the information encoding of original text data, the function is defined:
[0089]
[0090] in, Measures the clustering of a given data set The data coding rate changes, It measures the partition of protected attributes of given data The change of coding rate, represents the input data set, Θ represents the deep network parameters;
[0091] It is expected to obtain a data transformation function F(X; Θ) that can transform the original data into a more representative and fair feature space. The loss function is defined as:
[0092]
[0093] The former expects the encoding rate of each cluster subset to be as small as possible to achieve class aggregation, and the latter expects to satisfy the fairness definition. When the loss function is reduced to a specified threshold or the training rounds reach a given number, the expected fair representation space model is obtained.
[0094] It can be seen from the technical solutions provided by the above embodiments of the present invention that the method of the present invention can effectively cope with the complex challenges brought by limited high-dimensional data and unknown distributions, and can also accurately measure the encoding length of a given data representation, thereby ensuring that the data can maintain the integrity and accuracy of its original information when converted to the embedding space. This helps the model to better capture the intrinsic structure and characteristics of the data and improve the generalization ability and robustness of the model. This is particularly important for application scenarios involving sensitive data (such as financial credit, medical diagnosis, etc.). The method of the present invention can avoid unfairness caused by data bias and enhance users' trust and acceptance of the model.
[0095] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0097] Figure 1 A processing flow chart of a fair representation learning method for text data provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0098] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0099] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when the present invention refers to an element being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.
[0100] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
[0101] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.
[0102] The present invention proposes a fair representation learning method that integrates a non-progressive rate coding distortion metric function and an embedding space normalization technique to convert raw data into a more representative and fair feature space. The present invention uses a non-progressive rate distortion metric function to quantify data representation, ensuring that the integrity and accuracy of the original data can be maintained when converted to the feature space. This method is not only applicable to limited high-dimensional data, but also can effectively cope with the challenges brought by unknown distributions. By introducing this function, the present invention can more accurately measure the coding length of a given data representation, thereby ensuring that the representation of the data in the feature space has higher information content and semantic discriminability. The present invention uses an embedding space normalization technique to optimize the distribution and structure of the feature space. By rearranging and combining data points, the protected attributes become less important in the embedding space, thereby weakening their influence on the prediction results.
[0103] In order to better eliminate possible data bias in the learned representation and alleviate discrimination or unfairness in the decision-making of machine learning algorithms, the present invention uses a coding length function to evaluate the learned representation in the constructed embedding space, and the non-progressive distortion metric used can effectively deal with the unknown distribution and high-dimensional characteristics of given limited data.
[0104] The processing flow of a fair representation learning method for text data provided by an embodiment of the present invention is as follows: Figure 1 As shown, the processing steps include the following:
[0105] Step S1: Convert the collected text into a vocabulary, and use the vocabulary to construct a vocabulary embedding matrix consisting of discrete vectors.
[0106] In order to meet the specific needs of enterprises or universities in terms of demographic group segmentation, the present invention conducted a systematic data collection work through extensive online searches, screening and collecting a batch of online biographies written in English. During the data collection process, the present invention paid special attention to key information in the biographies, including names, pronouns, and occupations, to ensure the accuracy and relevance of the data. Through in-depth analysis of these biographies, the present invention can understand the gender distribution in different occupations and whether there is potential gender bias.
[0107] In order to convert the collected online biographical texts into numerical representations that can be efficiently processed by computer networks, the present invention uses a word list (also called a vocabulary or dictionary) to perform this conversion task. A word list is a list of all unique words in a text, where each word corresponds to a unique index. The present invention first preprocesses the text, including word segmentation, removal of stop words (such as common pronouns, conjunctions, etc.) and punctuation, and morphological restoration or stem extraction to ensure the conciseness and accuracy of the text information.
[0108] Then, the present invention constructs a comprehensive vocabulary based on the processed text, which contains all the unique words that appear. Next, the present invention uses this vocabulary to map each word in the text to a discrete vector, which is usually called one-hot encoding or binary encoding. In this encoding method, each vector has the same dimension as the number of words in the vocabulary, and only one position has a value of 1, and the values of the remaining positions are all 0. The position with a value of 1 corresponds to the index of the word in the vocabulary. In this way, the present invention converts the text into a vocabulary embedding matrix composed of discrete vectors, in which each row represents a sentence or document, and each column represents a word in the vocabulary. This representation method enables computers to easily perform numerical calculations and analysis on text, thereby supporting the application of various natural language processing and machine learning algorithms.
[0109] Step S2: Use BERT to convert the vocabulary embedding matrix into an embedding representation vector.
[0110] In the process of data processing, a deep neural network can be used to map the input sentence to the embedding space to obtain the embedded representation vector of the sentence. The deep neural network of the present invention adopts BERT (Bidirectional Encoder Representations from Transformers), which is a neural network language model based on Transformer. The embedded representation vector of the sentence of the BERT network is expressed by the following formula:
[0111] Z=F([CLS]+X)
[0112] Where Z is the sentence embedding representation vector, F(~) is the pre-trained deep neural network model (BERT in this case), [CLS] is a special marker symbol used to represent the overall semantics of the sentence, + represents the concatenation operation of the vector, and X is the pre-processed input sentence, that is, the vocabulary embedding matrix after conversion in step 1.
[0113] In this formula, the input sentence is first concatenated with a special marker symbol [CLS] and then input into the pre-trained deep neural network model. The model output is the sentence embedding representation vector Z = F(X; Θ). This embedding representation vector Z retains the semantic and contextual information of the sentence and has high expressive power.
[0114] Step S3: Design a non-progressive distortion metric function based on the embedded representation vector Z = F(X; Θ), and use the non-progressive distortion metric function to measure the impact of protected attributes and cluster structures on the information encoding of the original text data through a fair representation learning algorithm. Design a corresponding loss function based on the measurement results, and use the loss function to train a deep neural network (BERT in this case) to ensure that the obtained embedded representation vector meets the fairness constraint. Construct a fair representation space model based on the trained deep neural network.
[0115] Specifically, in the above, the present invention introduces the general form of a fair representation learning method based on a metric function, in which the loss function related to the target task is and the fairness regularizer Playing a central role, the parameter λ is used to weigh the trade-off between fairness and utility:
[0116]
[0117] In previous methods, these two terms are usually designed as independent function terms to deal with different optimization objectives. For example, the loss function The cross entropy loss is often chosen to maximize the performance of the model on the target task; while the fairness regularizer It is often designed as the maximum average difference distance, etc., aiming to reduce the differences between different sensitive attribute groups to achieve fairness. Although this mechanism is effective, it is difficult to find a unified trade-off standard because different metric functions may have different measurement scales and priorities. The uncertainty of this trade-off may cause the model to deviate or become unstable during the optimization process.
[0118] To this end, the present invention attempts to evaluate the learned representation space in a more unified way. By constructing a fair representation learning algorithm within a framework, the present invention can focus on optimizing a unified metric, simplify the implementation and expansion process of the model, and avoid trade-offs and compromises between multiple metric functions. The algorithm flow is shown in Table 1.
[0119]
[0120] Specifically, the present invention hopes to obtain a data transformation function F(X; Θ) that can transform the original data into a more representative and fair feature space. Therefore, the present invention first needs to evaluate the quality of the transformed representation. Without loss of generality, given a data set Where x represents the data feature used in model training, y and s represent the corresponding data label and protected attribute value respectively. In the case of ε coding error, the present invention introduces the coding length function to accurately estimate the average number of coding bits required for data recoding:
[0121]
[0122] in, is the embedding representation after data transformation, D is the embedding representation dimension, and N is the number of samples. In order to extract information related to the data label y, the present invention estimates the encoding of samples of different categories separately. Inspired by the local popularity hypothesis, that is, samples of the same type are often located in the same popular subspace, the present invention divides the embedding representation Z into multiple data subsets Z = Z1∪Z2∪...∪Z K , each subset is located in a low-dimensional subspace. Therefore, for each data subset, the present invention can measure the average number of codes required to encode samples of different classes:
[0123]
[0124] in, is a set of diagonal matrices, whose diagonal elements represent the membership relationship between N samples and K clusters, that is, Π i (j, j) represents the probability that sample j belongs to cluster i. Similarly, the present invention can estimate the average number of codes required for sample subsets with different protected attributes:
[0125]
[0126] Among them, Z i is the data subset of the i-th class, N i is the number of samples contained in the i-th data subset, is the diagonal matrix of membership of different protected attributes corresponding to the data subset, Indicates whether sample j contains the protected attribute m.
[0127] Ideally, the learned features of the same cluster should be highly correlated, while the features of different clusters should be as inconsistent as possible. Therefore, from the perspective of encoding length, the average encoding rate of the entire set Z should be as large as possible to obtain better generalization performance, while the average encoding rate of each category subset should be as small as possible to achieve class aggregation:
[0128]
[0129] In addition, considering the possible attribute discrimination or bias in the data, the present invention is inspired by the adversarial network and attempts to confuse the protected attribute sample subsets in different category subsets. In other words, if the protected attributes in the representation can be confused, that is, different attributes do not affect the encoding rate of the category subset, the desired fair representation can be obtained. To this end, the present invention defines the fairness loss as follows:
[0130]
[0131] in, is the data subset of the i-th class, N i is the number of samples contained in the i-th data subset, is the diagonal matrix of membership of different protected attributes corresponding to the i-th data subset, Represents whether sample j contains the protected attribute m. By minimizing this fairness loss, a fair embedding representation can be obtained.
[0132] Different from the adversarial-based method, the present invention unifies the two loss functions in a framework and optimizes them collaboratively, as shown in Table 1, effectively avoiding the instability and limitations that may be caused by the adversarial strategy-based method.
[0133] Step 4: After cleaning, the text data to be processed is input into the fair representation space model, and the fair representation space model outputs a fair representation of the text data to be processed.
[0134] After successfully constructing the fair representation space model, the present invention adopts the same strategy as the training data to process the test text data. Specifically, the following steps are included: 1. Data cleaning: segmenting the test data, removing stop words and punctuation marks, and restoring the word form or extracting stems, etc., to ensure the conciseness and accuracy of the text information; 2. Feature extraction: After completing the processing of the test data, the present invention inputs the processed test data into the constructed fair representation space model. The fair representation space model will parse and transform the test data based on the patterns and rules learned in the training phase, and finally generate a fair representation corresponding to the test data. These fair representations will serve as inputs for subsequent analysis or decision-making processes, providing a fair and accurate basis for data-driven decision-making processes.
[0135] In summary, the embodiments of the present invention utilize the non-progressive nature of the coding length function to design a new fairness representation space construction method, which can effectively cope with the complex challenges brought by limited high-dimensional data and unknown distributions, and accurately measure the coding length of a given data representation, thereby ensuring that the data can maintain the integrity and accuracy of its original information when converted to the embedding space.
[0136] This paper introduces a cutting-edge embedding space normalization technology, which can better optimize the distribution and structure of the feature space. Based on the idea of adversarial networks confusing protected attributes, this paper further normalizes the distribution of sample points of different protected attributes in the embedding space, thereby weakening their influence on the prediction results. At the same time, the overall model can be optimized and adjusted within a unified framework, effectively avoiding the instability and limitations that may be caused by adversarial strategy-based methods.
[0137] This paper proposes a fair representation learning method for text data, which cleverly introduces the non-progressive rate coding distortion metric function into the framework of fair learning. This strategy can not only effectively cope with the complex challenges brought by limited high-dimensional data and unknown distribution, but also accurately measure the encoding length of a given data representation, thereby ensuring that the data can maintain the integrity and accuracy of its original information when converted to the embedding space. This helps the model better capture the intrinsic structure and characteristics of the data and improves the generalization ability and robustness of the model.
[0138] Secondly, in view of the potential correlation between protected attributes and prediction tasks, this paper introduces a cutting-edge embedding space normalization technology. By carefully designing and optimizing the distribution and structure of the embedding space, this technology can significantly weaken the impact of protected attributes on the model prediction results, ensuring model performance while improving the fairness of model decisions. This is especially important for application scenarios involving sensitive data (such as financial credit, medical diagnosis, etc.), which can avoid unfair phenomena caused by data bias and enhance users' trust and acceptance of the model.
[0139] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.
[0140] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.
[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0142] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A fair representation learning method for text data, characterized in that: include: Convert the collected text data into a vocabulary, and use the vocabulary to construct a vocabulary embedding matrix consisting of discrete vectors; Using BERT to convert the vocabulary embedding matrix into an embedding representation vector; Measuring the influence of protected attributes and cluster structures on the information encoding of original text data according to the embedded representation vector, designing a loss function based on the measurement result of the non-progressive distortion measurement function, training a deep neural network using the loss function, and constructing a fair representation space model based on the trained deep neural network; The text data to be processed is cleaned and then input into the fair representation space model, and the fair representation space model outputs a fair representation of the text data to be processed.
2. The method according to claim 1, characterized in that The method of converting the collected text data into a vocabulary and using the vocabulary to construct a vocabulary embedding matrix composed of discrete vectors includes: The collected online biographical text data are preprocessed, including word segmentation, removal of stop words and punctuation marks, and lemma restoration or stemming. A vocabulary is constructed based on the preprocessed text, and the vocabulary is used to map each word in the text data into a discrete vector. Each vector has the same dimension as the number of words in the vocabulary, and only one position has a value of 1, and the values of the remaining positions are 0. The position with a value of 1 corresponds to the index of the word in the vocabulary, thereby converting the text data into a vocabulary embedding matrix composed of discrete vectors, in which each row represents a sentence and each column represents a word in the vocabulary.
3. The method according to claim 2, characterized in that The method of using BERT to convert the vocabulary embedding matrix into an embedding representation vector includes: The bidirectional encoder representation technology BERT network is used to map the sentences in the vocabulary embedding matrix to the embedding space to obtain the embedding representation vector of the sentence. The embedding representation vector of the sentence of the BERT network is expressed by the following formula: Z=F([CLS]+X) Wherein, Z is the embedding representation vector of the sentence, F(~) is the pre-trained deep neural network model, [CLS] is a marking symbol used to represent the overall semantics of the sentence, + represents the concatenation operation of the vector, and X is the sentence in the vocabulary embedding matrix; By first concatenating the input sentence with a special marker symbol [CLS] and then inputting it into the pre-trained deep neural network model, the model output is the sentence embedding representation vector Z = F(X; Θ). This embedding representation vector Z retains the semantic and contextual information of the sentence.
4. The method according to claim 3, characterized in that The measuring the influence of the protected attribute and the cluster structure on the original text data information encoding according to the embedded representation vector includes: The transformed embedding representation vector Z is evaluated using a non-progressive rate-distortion metric function. Without loss of generality, given a dataset Where x represents the data features used in model training, y and s represent the corresponding data labels and protected attribute values, respectively. ε is set as the encoding error, and the average number of bits required to re-encode the input data is estimated: in, is the embedded representation after data transformation, D is the embedding representation dimension, and N is the number of samples; According to the cluster structure, the embedding representation vector Z is divided into multiple data subsets Z = Z1∪Z2∪…∪Z K , each subset is located in a low-dimensional subspace. For each data subset, the average number of codes required to encode the subset samples is measured: in, is a set of diagonal matrices, whose diagonal elements represent the membership relationship between N samples and K clusters, that is, Π i (j,j)∈[0,1] represents the probability that sample j belongs to cluster i; Based on the protected attribute values of the samples, the average number of codes required to encode samples with different protected attributes in the same cluster subset is estimated: Among them, Z i is the data subset of the i-th class, N i is the number of samples contained in the i-th data subset, is the diagonal matrix of membership of different protected attributes corresponding to the data subset, Indicates whether sample j contains the protected attribute m; The non-progressive rate characteristic of the non-progressive distortion metric function can be used to evaluate the information coding quality within a given ε range.
5. The method according to claim 4, characterized in that The method of designing a loss function based on the measurement result of the non-progressive distortion measurement function, training a deep neural network using the loss function, and constructing a fair representation space model based on the trained deep neural network includes: A data transformation function F(X; Θ) is obtained, which can transform the original data into a more representative and fair feature space, ensuring that the obtained embedded representation vector satisfies the fairness constraint, which is defined as follows: R(Z,ε)-R(Z,ε|Π s )=0 Among them, R(Z,ε) and R(Z,ε|Π s ) respectively measure the average number of codes required to encode the entire text data of the input and samples with different protected attributes, Z = F(X; Θ) represents the obtained embedding representation, ε is the allowed encoding error, is the corresponding protected attribute membership diagonal matrix, represents whether sample j contains the protected attribute m. The fairness constraint requires that the protected attribute does not affect the encoding rate of the data. If the fairness constraint can be met, the obtained embedding representation vector Z is considered fair; In order to measure the impact of protected attributes and cluster structures on the information encoding of original text data, the function is defined: in, Measures the clustering of given data The data coding rate changes, It measures the partition of protected attributes of given data The change of coding rate, represents the input data set, Θ represents the deep network parameters; It is expected to obtain a data transformation function F(X; Θ) that can transform the original data into a more representative and fair feature space. The loss function is defined as: The former expects the encoding rate of each cluster subset to be as small as possible to achieve class aggregation, and the latter expects to satisfy the fairness definition. When the loss function is reduced to a specified threshold or the training rounds reach a given number, the expected fair representation space model is obtained.
