Large-scale language model debiasing method, system, device and storage medium

Abstract

The present invention provides a large-scale language model debiasing method, system, device and storage medium, including: obtaining a large-scale pre-trained language model based on a neural network; using a data self-generation method to obtain text data for gender bias analysis for the large-scale pre-trained language model to form a corpus, and establishing a series of templates based on the corpus; using the template to calculate the gender bias level of the large-scale pre-trained language model, and calculating the correction gradient of the large-scale pre-trained language model according to the gender bias level by back propagation; updating the model parameters in the large-scale pre-trained language model according to the correction gradient, repeating the optimization update until the preset update completion condition is met, and obtaining a large-scale language model without gender bias. Therefore, the present invention analyzes the gender bias of the language model from the perspective of information theory, and regards the bias correction problem as a multi-objective optimization problem to reduce bias, and adopts an automatic generation algorithm to solve the problem of insufficient training data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large-scale language model debiasing method, system, device and storage medium. Background Art

[0002] Large-scale pre-trained language models have been widely used in various NLP (natural language processing) tasks and have achieved good results. However, many works have shown that these large-scale language models contain gender biases. Therefore, by using large-scale pre-trained models, these gender biases enter various downstream tasks, which may bring unpredictable risks to the system.

[0003] There are a range of approaches to de-gendering static word embeddings, such as data manipulation (Zhao et al., 2018b; Madaan et al., 2018; Prates et al., 2020; Vanmassenhove et al., 2019; Park et al., 2018), identifying gender subspaces (Schmidt, 2015; Bolukbasi et al., 2016), and designing new objective functions (Zhao et al., 2018a).

[0004] However, there are many difficulties in how to remove bias in large-scale pre-trained language models. First, because the original training data may not be public and training these models requires a lot of computing resources, it is difficult to reduce gender bias in the model through data manipulation or new objective functions. Second, unlike static word embeddings, a small change in a large-scale pre-trained model may cause great damage to the model. In this case, it is urgent to find an efficient, low-cost, and training data-independent debiasing algorithm. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a large-scale language model debiasing method, system, device and storage medium. The present invention provides an efficient, low-cost debiasing algorithm that is independent of training data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A large-scale language model debiasing method, comprising:

[0008] Get large-scale pre-trained language models based on neural networks;

[0009] A data self-generation method is used to obtain text data for gender bias analysis using the large-scale pre-trained language model to form a corpus, and a series of templates are established based on the corpus;

[0010] Calculating the gender bias level of the large-scale pre-trained language model using the template, and calculating the correction gradient of the large-scale pre-trained language model by back-propagation according to the gender bias level;

[0011] The model parameters in the large-scale pre-trained language model are updated according to the correction gradient, and the optimization update is repeated until the preset update completion conditions are met to obtain a large-scale language model without gender bias.

[0012] As a further improvement of the present invention, the large-scale pre-trained language model is a GPT-2 model, in which the natural language corpus is represented as a symbol sequence [ s 1 ,s 2 ,...,s n+1 ]; large-scale pre-trained language models are modeled as the task of computing conditional probabilities:

[0013] p ( s n+1 |s 1 ,s 2 ,...,s n ) . (1)

[0014] Large-scale pre-trained language models consist of two parts: the first part is token embedding ε The symbol sequence is converted into an embedding vector sequence for the calculation of the neural network, that is,

[0015]

[0016] In particular, for static token embedding ε have:

[0017]

[0018] The other part is the neural network structure used to calculate the conditional probability (1) M for:

[0019] .

[0020] As a further improvement of the present invention, the method of using a data self-generation method for the large-scale pre-trained language model to obtain text data for gender bias analysis specifically includes:

[0021] For the data to be used for deviation calculation, two requirements must be met:

[0022] 1) The context should link the concepts of occupation and gender. The text data must contain the target occupation words and meet the following requirements:

[0023]

[0024] For some threshold s;

[0025] 2) Text data should not contain information that indicates gender and must consist of neutral words.

[0026] As a further improvement of the present invention, the large-scale pre-trained language model is used to obtain text data for gender bias analysis using a data self-generation method to form a corpus, and a series of templates are established based on the corpus, specifically including:

[0027] For the selected occupation, determine the beginning of the sentence;

[0028] Using the prediction results of the GPT-2 model, randomly add words one by one to the generated data to generate sentences;

[0029] In this process, neutrality is used as the criterion for word selection, and the middle is used as the criterion for the end of the algorithm; in the process of judging whether a word is related to gender, the list of words contributed by is used;

[0030] If both requirements are met, the sentences that meet the requirements are used as a corpus, and a series of templates are established based on the corpus and saved.

[0031] As a further improvement of the present invention, the use of the template to calculate the gender bias level of the large-scale pre-trained language model specifically uses the gain of gender information to measure the gender bias of the model in a given context, which is expressed as:

[0032] Bias ( Context) = H (Predicted gender in semantics) -H (Gender predicted by the model) , (3)

[0033] in H ( . ) is a function of Shannon entropy;

[0034] The model's gender prediction is calculated using pronoun choice, i.e.:

[0035]

[0036] The expected gender bias of all text data containing a certain occupation is regarded as the gender bias of the model for a certain occupation, that is:

[0037] Bias ( ε,M, Occupation) = ( Bias ( ε,M, context)), for all contexts Occupation(4)

[0038] And define the gender bias of the model in the occupational dimension, using bias It is the average bias of all occupations.

[0039] As a further improvement of the present invention, updating the model parameters in the large-scale pre-trained language model according to the correction gradient specifically includes:

[0040] Accuracy tag embedding tuning uses self-supervised learning to find an approximate solution to the following bi-objective optimization problem

[0041]

[0042] Where α is a hyperparameter;

[0043] When calculating specifically, it is decomposed into several sub-problems, and the gradient descent method is used to solve each sub-problem, namely

[0044] (8)

[0045] in V o is a word list of all occupations; for the first and, The solution is obviously ; For the second sum, use the Adam algorithm to minimize ;

[0046] Finally, the set of solutions to all subproblems is used as the debiased embedding parameters; the average of all embedding parameters is used as the final result.

[0047] As a further improvement of the present invention, after obtaining the large-scale language model without gender bias, a multi-objective optimization step is further included, specifically:

[0048] Using perplexity As an indicator of model performance, the debiasing problem is expressed as:

[0049] (5).

[0050] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the large-scale language model debiasing method when executing the computer program.

[0051] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the large-scale language model debiasing method.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] The present invention provides a debiasing method for overcoming the above difficulties on a large-scale language model. A data self-generation method is used for a large-scale pre-trained language model to obtain text data for gender bias analysis, form a corpus, and establish a series of templates based on the corpus; the gender bias level is calculated according to the template, and the correction gradient of the large-scale pre-trained language model is calculated by back propagation according to the gender bias level; then the model parameters in the large-scale pre-trained language model are updated according to the correction gradient to obtain a large-scale language model without gender bias, which can prevent the problem of gender discrimination caused by the use of existing large-scale language models. By designing a suitable loss function, the present invention can significantly reduce the gender bias in GPT-2 models of different scales without destroying the model, which is actually a multi-objective optimization. Therefore, the present invention analyzes the gender bias of the language model from the perspective of information theory, and regards the bias correction problem as a multi-objective optimization problem to reduce the bias, and adopts an automatic generation algorithm to solve the problem of insufficient training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of the large-scale language model debiasing method of the present invention;

[0055] Figure 2 A schematic diagram showing how people and language models make different judgments about gender based on the same text data;

[0056] Figure 3 This is an illustration of the situation in which gender bias is measured in the context of “nurses say” in this example;

[0057] Figure 4 For selected career embodiments;

[0058] Figure 5 This is a flow chart of using the gradient descent method to solve each sub-problem;

[0059] Figure 6 This is a comparison chart of the changes in gender bias and perplexity of the model before and after debiasing;

[0060] Figure 7 It is a schematic diagram of the structure of the large-scale language model debiasing system of the present invention;

[0061] Figure 8 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0062] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0063] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0064] It should be noted that the terminals involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers, personal computers (PCs), MP3 players, MP4 players, wearable devices (for example, smart glasses, smart watches, smart bracelets, etc.), smart home devices and other smart devices.

[0065] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0066] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention.

[0067] like Figure 1 As shown, the present invention provides a large-scale language model debiasing method, comprising:

[0068] Get large-scale pre-trained language models based on neural networks;

[0069] A data self-generation method is used to obtain text data for gender bias analysis using the large-scale pre-trained language model to form a corpus, and a series of templates are established based on the corpus;

[0070] Calculating the gender bias level of the large-scale pre-trained language model using the template, and calculating the correction gradient of the large-scale pre-trained language model by back-propagation according to the gender bias level;

[0071] The model parameters in the large-scale pre-trained language model are updated according to the correction gradient, and the optimization update is repeated until the preset update completion conditions are met to obtain a large-scale language model without gender bias.

[0072] Gender bias in large-scale language models has attracted increasing academic interest. The present invention provides a gender bias mitigation method with limited impact on the performance of the language model, using a self-supervised learning method to determine a precision strategy for adjusting token embeddings, which can reduce the bias of token embeddings in the gender subspace while minimizing side effects in the gender subspace. The method of the present invention is used to mitigate the gender bias in the GPT-2 model. Experimental results show that the method of the present invention can find a debiasing strategy with limited impact on the model prediction performance.

[0073] By designing a suitable loss function, the present invention can significantly reduce the gender bias in GPT-2 models of different scales without destroying the model, which is actually a multi-objective optimization.

[0074] The specific contents of the present invention are described below with reference to specific embodiments.

[0075] The first is bias analysis. Current work describes gender bias in language models from different perspectives. The first idea comes from psychology. Using the concept of implicit association test, scholars measure gender bias by the difference in the strength of concept associations. The second idea is to define gender bias as the projection of gender-neutral word embeddings into the gender subspace. The first idea is to analyze the gender bias of language models from NLP tasks. In particular, most contextualized models can directly complete some specific NLP tasks, and a large amount of work directly measures gender bias through the predictions of contextualized models.

[0076] Possible debiasing methods,To mitigate LM bias, common approaches include modifying the training data through data augmentation,,manipulating word embeddings, and adjusting predictions to produce a fairer language model.,While much has been studied in static word embeddings, some recent work,studies gender bias methods in contextualized models.

[0077] In language models, natural language corpora are represented as sequences of symbols [ s 1 ,s 2 ,...,s n+1 ] in the form of . Therefore, language models are usually modeled as the task of calculating conditional probabilities:

[0078] p ( s n+1 |s 1 ,s 2 ,...,s n ) . (1)

[0079] This method is based on pre-trained language models of neural networks. Most of these models consist of two parts. The first part of the model is the token embedding ε The symbol sequence is converted into an embedding vector sequence for the calculation of the neural network, that is,

[0080]

[0081] In particular, the present invention assumes that the embedding method is static, i.e.

[0082]

[0083] The other part of the model is a neural network structure used to calculate the conditional probability (1) M , such as the converter, that is

[0084]

[0085] Figure 2 Based on the same text data, people and language models will make different judgments on gender information. The root cause is that the gender bias of the language model obtains the gender information of the text data.

[0086] Information gain and gender bias analysis, a lot of evidence shows that language models predict gender differently from human semantic judgments. Figure 2 As shown in Figure 1, the main difference is that the model predicts that gender information has lower entropy. This is because the language model has gender bias, so gender information is added based on the input text data. Therefore, the gain of gender information can be used to measure the gender bias of the model for a given context, which is expressed as

[0087] Bias ( Context) = H (Predicted gender in semantics) -H (Gender predicted by the model) , (3)

[0088] in H ( . ) is a function of Shannon entropy.

[0089] In the above formula, the model's pronoun choice is used to calculate the model's gender prediction, that is,

[0090]

[0091] exist Figure 3 In the present invention, the context "nurse said" is used as an example to calculate gender bias.

[0092] Figure 3 : In this example, the present invention provides a situation where gender bias is measured in the instance context = "the nurse said". Note that the model outputs a much higher probability of "he" than "she", which results in low entropy of gender information.

[0093] In addition, the model's expectation of the gender bias of all text data containing a certain occupation is regarded as the model's gender bias for a certain occupation, that is,

[0094] Bias ( ε,M, Occupation) = ( Bias ( ε,M, context)), for all contexts Occupation(4)

[0095] And define the gender bias of the model in the occupational dimension, using bias It is the average bias of all occupations.

[0096] Multi-objective optimization,When dealing with the gender bias problem in language models, it is not enough to only focus on the value of the bias.,For example, a uniform distribution over the vocabulary must be an unbiased model, but models such as language models are meaningless.,Thus, the bias correction problem. It has at least two dimensions, the bias of the model and the protection of the model.

[0097] If the present invention uses "perplexity" Perplexity As an indicator of model performance, the debiasing problem can be expressed as

[0098] (5).

[0099] The self-supervised accuracy debiasing method is described as follows:

[0100] Data self-generation method,In the definition of gender bias (3), the measurement of bias requires some specific text data. Currently, there are some datasets that can provide similar corpora for gender bias analysis. However, in most existing datasets, due to the method of manually designing language templates, the amount of data is relatively small, which makes it difficult to support the debiasing work in large-scale language models. Therefore, the present invention hopes to obtain sufficient data through automatic data generation.

[0101] For the data used for deviation calculation, the present invention proposes two requirements:

[0102] Intermediary: The context should link the two concepts of occupation and gender. In this regard, the present invention requires that the text data contains the target occupation words and satisfies

[0103]

[0104] Among them, s is the threshold.

[0105] Neutral: Text data should not contain information indicating gender. Therefore, the present invention requires that text data be composed of neutral characters.

[0106] Figure 4 For the selected occupation, the invention determines the beginning of the sentence "the [occupation]". Then, the words are added one by one through GPT-2 to generate the sentence. In this process, the invention uses neutral as the criterion for word selection and middle as the criterion for the end of the algorithm. In the process of determining whether the word is related to gender, the invention uses the word list contributed by.

[0107] This list can be obtained from an existing database, such as https: / / github.com / uclanlp / gn glove. And on the right is an example of generating a metric about the gender bias of doctors, where the threshold s is selected as 0.1. In the fourth row, because girl is related to gender, the words are reselected.

[0108] In view of these two points, a data generation algorithm based on the GPT-2 text data generation task is designed, such as Figure 4 In practice, in order to ensure efficiency, the present invention limits the length of a sentence to no more than 15 words.

[0109] Regarding the precision mark embedding adjustment, we return to the optimization problem (5). Undoubtedly, problem (5) is an extremely complex problem. In order to reduce the computational difficulty, the present invention reduces the parameters to be optimized and considers the approximation problem.

[0110] (6)

[0111] in are the parameters of the pre-trained model. Even so, Problem 6 is still complex enough. Fortunately, the pre-trained model has been fully trained, and the present invention can assume that

[0112]

[0113] In addition, the present invention assumes that on a given dataset, the perplexity of the model is always Here, the present invention uses self-supervised learning to find an approximate solution to the following dual-objective optimization problem:

[0114]

[0115] where α is a hyperparameter.

[0116] In the debiasing work, solving problem 7 has two advantages. First, the optimization objective ensures the reduction of model bias. Second, problem 7 can be decomposed into several sub-problems to further reduce the computational difficulty, namely

[0117] (8)

[0118] in, V o is a word list of all occupations. For the first and, The solution is obviously For the second sum, the present invention uses the Adam algorithm to minimize Finally, the present invention uses the set of solutions to all sub-problems as the solution to the problem. The whole process is shown in FIG. 5 Shown in.

[0119] Figure 5 The debiasing problem is divided into several sub-problems, and the gradient descent method is used to solve each sub-problem. Finally, the solutions of each sub-problem are collected into the debiased embedding parameters.

[0120] In fact, due to byte pair encoding, there may be some identical tokens in the token representations of different occupation words. In this case, the present invention uses the average value of the embeddings of all optimized tokens as the final result.

[0121] Simulation Verification

[0122] The present invention conducted debiasing experiments on four GPT-2 models of different sizes, namely gpt2 (117M parameters), gpt2-medium (345M parameters), gpt2-large (774M parameters), and gpt2-xl (1558M parameters). The present invention used the pre-trained weights provided by the converter Python library. Using the previous algorithm, the present invention generated 500 training data and 10 test data for each of the 120 occupations. Then, the present invention used the algorithm of the present invention to debias the four models, where the hyperparameter a was taken as 1000. The changes in gender bias and perplexity of the model before and after debiasing are shown in Figure 2. Figure 5 As shown, the perplexity is calculated for the Wikitext-2 dataset.

[0123] Figure 6 The bias of each model is significantly reduced, and the perplexity of each model remains within an acceptable range after debiasing. In particular, on gpt2-large and gpt2-x1, the perplexity of the debiased model of the present invention is almost the same as that of the original model. This means that the method of the present invention has achieved great success in these two models.

[0124] In summary, a new metric is designed based on information theory of gender bias in language models, and the protection of model performance is incorporated into the debiasing goal. In this framework, a self-supervised learning method is designed to adjust the token embedding and make full use of the knowledge learned during pre-training. The method of the present invention achieves excellent debiasing results on GPT-2. In addition, the method of the present invention can also be applied to other language models and other types of bias.

[0125] like Figure 7 As shown, the present invention also provides a large-scale language model debiasing system, comprising:

[0126] Model acquisition module, which acquires large-scale pre-trained language models based on neural networks;

[0127] A template establishment module, which uses a data self-generation method to obtain text data for gender bias analysis for the large-scale pre-trained language model to form a corpus, and establishes a series of templates based on the corpus;

[0128] A parameter calculation module, which uses the template to calculate the gender bias level of the large-scale pre-trained language model, and calculates the correction gradient of the large-scale pre-trained language model by back-propagation according to the gender bias level;

[0129] The parameter updating module updates the model parameters in the large-scale pre-trained language model according to the correction gradient, and repeatedly optimizes and updates until the preset update completion conditions are met to obtain a large-scale language model without gender bias.

[0130] like Figure 8 As shown, the third object of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the large-scale language model debiasing method when executing the computer program.

[0131] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0132] The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI (graphical user interface) on an external input / output device (such as a display device coupled to the interface). In other embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, each device providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0133] If necessary, it also has an input device for receiving input digital or character information, and generating key signal input related to user settings and function control of the electronic device that implements the training method of the language model provided in the embodiment of the present application, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 504 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, an LCD (liquid crystal display), an LED (light emitting diode) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0134] A fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large-scale language model debiasing method are implemented.

[0135] The large-scale language model debiasing method of the present invention can be widely used in the government's supervision of AI applications. On the one hand, it is used to ensure the fairness of artificial intelligence algorithms in government public projects; on the other hand, this method and technology can also be used for government supervision of the fairness of commercial artificial intelligence, and it can be forcibly embedded in all commercial artificial intelligence algorithms to ensure algorithm fairness. It can effectively prevent the problem of gender discrimination caused by the use of existing large-scale language models. For example, in the fields of human resources, legal proceedings, etc., it can effectively prevent the problem of gender discrimination caused by the application of artificial intelligence.

[0136] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, PLD (programmable logic device)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0137] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A large-scale language model debiasing method, characterized in that: include: Get large-scale pre-trained language models based on neural networks; A data self-generation method is used to obtain text data for gender bias analysis using the large-scale pre-trained language model to form a corpus, and a series of templates are established based on the corpus; Calculating the gender bias level of the large-scale pre-trained language model using the template, and calculating the correction gradient of the large-scale pre-trained language model by back-propagation according to the gender bias level; The model parameters in the large-scale pre-trained language model are updated according to the correction gradient, and the optimization update is repeated until the preset update completion conditions are met to obtain a large-scale language model without gender bias; The method of using a data self-generation method to obtain text data for gender bias analysis for the large-scale pre-trained language model specifically includes: For the data to be used for deviation calculation, two requirements must be met: 1) The context should link the concepts of occupation and gender. The text data must contain the target occupation words and meet the following requirements: For some threshold s; 2) Text data should not contain information that indicates gender, and should consist of neutral words; The method of using a data self-generation method to obtain text data for gender bias analysis for the large-scale pre-trained language model to form a corpus, and establishing a series of templates based on the corpus, specifically includes: For the selected occupation, determine the beginning of the sentence; Using the prediction results of the GPT-2 model, randomly add words one by one to the generated data to generate sentences; In this process, neutrality is used as the criterion for word selection, and the middle is used as the criterion for the end of the algorithm; in the process of judging whether a word is related to gender, the list of words contributed by is used; If both requirements are met, the sentences that meet the requirements are used as a corpus, and a series of templates are established based on the corpus and saved; The use of the template to calculate the gender bias level of the large-scale pre-trained language model specifically uses the gain of gender information to measure the gender bias of the model in a given context, which is expressed as: Bias ( Context) = H (Predicted gender in semantics) -H (Gender predicted by the model) , (3) in H ( . ) is a function of Shannon entropy; The model's gender prediction is calculated using pronoun choice, i.e.: The expected gender bias of all text data containing a certain occupation is regarded as the gender bias of the model for a certain occupation, that is: Bias ( ε,M, Occupation) = ( Bias ( ε,M, context)), for all contexts Occupation(4) And define the gender bias of the model in the occupational dimension, using bias It is the average bias of all occupations.

2. The large-scale language model debiasing method according to claim 1, characterized in that: The large-scale pre-trained language model is a GPT-2 model, in which the natural language corpus is represented as a sequence of symbols [ s 1 , s 2 ,...,s n+1 ]; large-scale pre-trained language models are modeled as the task of computing conditional probabilities: p ( s n+1 |s 1 ,s 2 ,...,s n ) . (1) Large-scale pre-trained language models consist of two parts: the first part is token embedding ε The symbol sequence is converted into an embedding vector sequence for the calculation of the neural network, that is, In particular, for static token embedding ε have: The other part is the neural network structure used to calculate the conditional probability (1) M for: 。 3. The large-scale language model debiasing method according to claim 1, characterized in that: The updating of the model parameters in the large-scale pre-trained language model according to the correction gradient specifically includes: Accuracy tag embedding tuning uses self-supervised learning to find an approximate solution to the following bi-objective optimization problem Where α is a hyperparameter; When calculating specifically, it is decomposed into several sub-problems, and the gradient descent method is used to solve each sub-problem, namely （8） in V o is a word list of all occupations; for the first and, The solution is obviously ; For the second sum, use the Adam algorithm to minimize ; Finally, the set of solutions to all subproblems is used as the debiased embedding parameters; the average of all embedding parameters is used as the final result.

4. The large-scale language model debiasing method according to claim 1, characterized in that: After obtaining the large-scale language model without gender bias, a multi-objective optimization step is also included, specifically: Using perplexity As an indicator of model performance, the debiasing problem is expressed as: （5）。 5. A large-scale language model debiasing system, based on the large-scale language model debiasing method according to any one of claims 1 to 4, characterized in that: include: Model acquisition module, which acquires large-scale pre-trained language models based on neural networks; A template establishment module, which uses a data self-generation method to obtain text data for gender bias analysis for the large-scale pre-trained language model to form a corpus, and establishes a series of templates based on the corpus; A parameter calculation module, which uses the template to calculate the gender bias level of the large-scale pre-trained language model, and calculates the correction gradient of the large-scale pre-trained language model by back-propagation according to the gender bias level; The parameter updating module updates the model parameters in the large-scale pre-trained language model according to the correction gradient, and repeatedly optimizes and updates until the preset update completion conditions are met to obtain a large-scale language model without gender bias.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the large-scale language model debiasing method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the large-scale language model debiasing method according to any one of claims 1 to 4.

Images