Pseudo-correlation relieving method and system based on machine forgetting

Abstract

The invention discloses a false correlation relieving method and system based on machine forgetting, and belongs to the technical field of deep learning and natural language processing. Aiming at the problems of insufficient generalization ability and limited application in high-risk scenes such as medicine, finance and the like caused by excessive dependence of a current model on false correlation between data and labels, the invention innovatively adopts a proxiity-based neuron positioning method to realize more accurate positioning of neurons. The neuron positioning module is responsible for analyzing a storage mechanism of the language model for causal and pseudo-correlation information and positioning causal or pseudo-correlation neurons. The neuron editing module is responsible for editing positioned neuron parameters so as to forget pseudo-correlation priori of the model, the method is verified on sentiment analysis and natural language reasoning tasks, and generalization of the model exceeds 3.13% and 2.04% of a baseline respectively.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning and natural language processing, and particularly relates to a method and system for alleviating pseudo-correlation based on machine forgetting. Background Art

[0002] Currently, data-driven language models (LMs), especially large language models (LLMs), have achieved state-of-the-art performance in a wide range of natural language processing tasks. However, recent research has shown that LMs lack out-of-distribution (OOD) generalization ability, and one of the main reasons is that these LMs capture the pseudo-correlation between training data and labels.

[0003] Currently, the strategies for alleviating pseudo-correlation mainly focus on data preprocessing or model regularization, and can be divided into three categories: preprocessing, inprocessing, and postprocessing. The preprocessing method retrains the model by removing the data containing pseudo-correlation in the training dataset or reducing its proportion, so as to correct the pseudo-correlation prior learned by the model. The inprocessing method effectively reduces the model's dependence on false-correlated data by adjusting the training strategy. The postprocessing method corrects the model output by eliminating the pseudo-correlation effect in the prediction stage. However, the above strategies have the following limitations: (1) Insufficient pseudo-correlation correction. The current strategies fail to directly edit the model parameters, so that the internalized pseudo-correlation prior in the model cannot be fully and accurately eliminated. (2) High cost. These strategies require carefully constructing a challenge dataset (i.e., data without pseudo-correlation) and retraining the model, with a relatively high cost.

[0004] Machine forgetting refers to the process by which a model partially or completely deletes the memory of specific data or information, and is commonly used in privacy protection research. Neuron parameter editing is an effective machine forgetting method that can fully and precisely delete the target information without retraining the model. Summary of the Invention

[0005] Aiming at the problem that the current model overly relies on the pseudo-correlation between data and labels, resulting in insufficient generalization ability, especially limited application in high-risk scenarios such as medicine and finance, the present invention provides a method and system for alleviating pseudo-correlation based on machine forgetting to improve the model's generalization.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for alleviating pseudo-correlation based on machine forgetting, comprising the following steps:

[0008] Step 1, locate the pseudo-correlated neurons by the gradient response varying with neurons based on proximity, where proximity is the degree of proximity between the set of correlation tokens extracted by the model and the causal token set or the pseudo-correlation token set;

[0009] Step 1.1, construct a gold token set to guide the positioning of neurons; including constructing a causal token set (gold causal token set) and a spurious correlation token set (gold spurious correlation token set);

[0010] Step 1.2, extract the correlation token set; analyze the attention scores of the output model for tokens through the attention attribution method, and extract the tokens with high attention scores to form the correlation token set;

[0011] Step 1.3, neuron positioning; calculate the gradient of proximity with respect to neurons and perform integral operations to accurately locate the neurons related to spurious correlation information or causal correlation information;

[0012] Step 2, adjust the neuron parameters based on the particle swarm optimization algorithm to adjust the model parameters to the optimal state; in this state, the model will forget the spurious correlation prior while striving to maintain the memory of causal correlation;

[0013] Step 2.1, classify the located neurons according to different editing purposes and methods, namely neurons that only store spurious correlation information, neurons that only store causal correlation information, and neurons where spurious correlation information and causal information overlap;

[0014] Step 2.2, design different editing methods based on different neuron categories, specifically: for neurons that only store spurious correlation information, set their parameters to zero; for neurons that only store causal correlation information, keep their parameters unchanged; for neurons where spurious correlation information and causal information overlap, edit them through a method based on the PSO algorithm.

[0015] Furthermore, in Step 1.1, construct a gold token set to guide the positioning of neurons; including constructing a causal token set (gold causal token set) and a spurious correlation token set (gold spurious correlation token set), the specific method is:

[0016] First, construct the causal token set (gold causal token set):

[0017] According to the prior knowledge of humans about the training data set, extract the tokens with strong causal relationships with the labels, and construct the causal token set (gold causal token set) denoted as Gcau ;

[0018]

[0019] Among them, label j is the j-th label in this task; is the set of tokens that have a causal relationship with label j ;

[0020] Then, construct a pseudo-correlation token set (gold spurious correlation token set), including misclassified tokens and irrelevant tokens;

[0021] For misclassified tokens, for the current label, misclassified tokens refer to tokens that have a causal relationship with other labels; Use G cau to quickly obtain the set of misclassified tokens label -j represents other labels except the j-th label; is the set of misclassified tokens related to label j ; is the set of causal tokens related to label -j ;

[0022] For irrelevant tokens, tokens that are significantly irrelevant to the label are defined as irrelevant tokens, and the set of irrelevant tokens is obtained

[0023] The pseudo-correlation token set (gold spurious correlation token set) is denoted as G spu ;

[0024]

[0025] Furthermore, in step 1.2, extraction of the correlation token set; Analyze the attention scores of the output model for tokens by the attention attribution method, and extract the tokens with high attention scores to form the correlation token set; The specific steps are as follows:

[0026] Step 1.2.1, output of attention scores: Assume that f is a trained model, given a corpus D, for each input example e i , require the model to output each token in example e i : The corresponding attention scores where m is the number of tokens in the input;

[0027] Step 1.2.2, Attention Ranking: Rank the attention scores for each example;

[0028] Step 1.2.3, Correlation Token Extraction: Extract the top Top-K tokens by attention ranking, and finally form a correlation token set correlation set on the entire corpus is a subset of correlation tokens related to the j-th label. Note: The relationship between the extracted Top-K related tokens and the model prediction may be causal correlation or spurious correlation. These spurious correlations need to be forgotten by the model.

[0029] Furthermore, in Step 1.3, Neuron Localization; by calculating the gradient of the proximity between the correlation token set and the causal token set or spurious correlation token set varying with neurons, an integration operation is performed to accurately locate the neurons related to spurious correlation information or causal correlation information, specifically including the following steps:

[0030] Step 1.3.1, Further Define the Correlation Set: C is the correlation set obtained by the language model on the dataset D, expressed as:

[0031]

[0032] where, represents the k-th neuron in the l-th layer of the model f, is the value of;

[0033] Step 1.3.2, The gradient integration of proximity is expressed as:

[0034]

[0035] Gradually change the value of from 0 to the original value obtained from the LM When α changes from 0 to 1, by integrating the gradient, accumulate the change in proximity caused by the change in ; if the neuron has a significant impact on proximity, then will be very large, indicating that the neuron stores the corresponding causal or spurious correlation information.

[0036] where, Indicates the gradient of the proximity between the correlation tokens extracted by the computational model and the set of causal or pseudo-correlation tokens with respect to variation; G ∈ (G cau , G spu ), if G = G cau , then this calculation formula is used to locate causal neurons, otherwise this formula is used to locate pseudo-correlation neurons;

[0037] Indicates the proximity between

[0038]

[0039] and G, expressed as: where represents the total attention score of the tokens in

[0040] Step 1.3.3, Integrate the above gradient to locate causal neurons and pseudo-correlation neurons.

[0041] Furthermore, in Step 2.2 of the above, for neurons where pseudo-correlation information and causal information overlap, a method based on the PSO algorithm is used to edit them. The specific method is:

[0042] Step 2.2.1, Particles and Population: Define the number of neurons as n, and the set of neuron values as x = (x 1 , x 2 , …, x n ), where x n is the value of the nth neuron; in the optimization, this group of neurons is regarded as a particle, and multiple particles form a population X; each particle contains two pieces of information: position (x) and velocity (v). The position information is the parameter value of the neuron, and the velocity information describes the magnitude and direction of the change in the neuron value in the search space. These two pieces of information together determine the iterative value of the neuron;

[0043] Step 2.2.2, Evaluate the quality of the particle position through the fitness function fitness. The higher the fitness, the closer the particle position is to the optimal solution of the problem, expressed as:

[0044] fitness(x) = proximity(C (D,x) , G cau ) - proximity(C (D,x) , G spu )

[0045] where C (D,x)Refers to the correlationset obtained from the corpus D when the parameters of the model neuron are x, proximity(C (D,x) ,C cau ) is the degree of proximity between C and G cau , proximity(C (D,x) ,G spu ) is the degree of proximity between C and G spu ;

[0046] Step 2.2.3, Iterative mechanism: The iterative mechanism combines individual learning and group learning; The particle updates its own position and velocity according to the information of its own optimal position p i and the global optimal position g; The particle optimal position is the position with the highest fitness found by a single particle during the search process, and the global optimal position is the position with the highest fitness found by the entire particle swarm during the search process; The mechanism is as follows:

[0047] v i,t+1 =wv i,t +c 1 r 1 (p i,t -x i,t )+c 2 r 2 (g t -x i,t )

[0048] x i,t+1 =x i,t +v i,t+1

[0049] where, v i,t is the velocity of the i-th particle in the t-th iteration, w is the inertia weight, used to balance the influence of the particle's current velocity and past velocity; c 1 and c 2 are learning factors, used to adjust the learning speed; r 1 and r 2 are random numbers, used to introduce randomness; x i,t is the position of the i-th particle in the t-th iteration; By continuously iterating, the particle position x * that maximizes f can be obtained, that is, the optimal combination of neuron parameters;

[0050] x * =argmax x fitness(x)

[0051] Stopping criterion: Set the number of iterations to 50, and when this upper limit is reached, the optimization algorithm stops iterating; Based on the above settings, optimize the parameters of the neuron and finally reach the optimal state.

[0052] A machine forgetting-based pseudo-relevance mitigation system according to the above method, the system comprising: a neuron localization module, a neuron editing module; the neuron localization module is a proximity-based neuron localization module, responsible for accurately identifying neurons related to pseudo-relevant information; the neuron editing module, responsible for editing the parameters of the located neurons to forget the pseudo-relevant prior of the model.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] The present invention introduces machine forgetting into the research of pseudo-relevance mitigation tasks, with great innovation and application value. The method of the present invention is realized through a neuron localization module (i.e., proximity-based neuron localization) and a neuron editing module based on the PSO algorithm, and has the characteristics of low cost and more sufficient correction ability. This method not only significantly reduces the model's dependence on pseudo-relevance, but also strives to maintain its memory of causal relationships. The evaluation performance on sentiment analysis and natural language inference tasks exceeds the fine-tuning method by 3.13% and 2.04%. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 It is an architecture diagram of a machine forgetting-based language model pseudo-relevance mitigation strategy proposed by the present invention;

[0057] Figure 2 It is a flowchart of a proximity-based neuron localization module;

[0058] Figure 3 It is a flowchart of a neuron editing module based on the particle swarm optimization algorithm. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] In order to deeply understand the present invention, we will describe it comprehensively and meticulously. However, the present invention has various implementation manners and is not limited to the specific examples listed herein. The presentation of these examples aims to deepen the comprehensive understanding of the disclosed content of the present invention.

[0060] Embodiment 1

[0061] A method for machine forgetting-based pseudo-relevance mitigation, comprising the following steps:

[0062] Step 1, locate pseudo-correlated neurons by the gradient response that varies with neurons based on proximity, where proximity is the degree of closeness between the set of correlation tokens extracted by the model and the causal token set or the pseudo-correlation token set;

[0063] Step 1.1, construct a set of gold tokens to guide the location of neurons; including constructing a causal token set (gold causal token set) and a pseudo-correlation token set (gold spurious correlation token set); the specific method is:

[0064] First, construct the causal token set (gold causal token set): According to the prior knowledge of humans about the training data set, extract the tokens that have a strong causal relationship with the label, and construct the causal token set (gold causal token set) denoted as G cau ;

[0065]

[0066] where label j is the j-th label in this task; is the set of tokens that have a causal relationship with label j ;

[0067] Then, construct the pseudo-correlation token set (gold spurious correlation token set), including misclassified tokens and irrelevant tokens;

[0068] For misclassified tokens, for the current label, misclassified tokens refer to the tokens that have a causal relationship with other labels; use G cau to quickly obtain the set of misclassified tokens label -j represents other labels except the j-th label; is the set of misclassified tokens related to label j ; is the set of causal tokens related to label -j ;

[0069] For irrelevant tokens, define the tokens that are significantly irrelevant to the label as irrelevant tokens, and obtain the set of irrelevant tokens

[0070] The gold spurious correlation token set is denoted as G spu ;

[0071]

[0072] Step 1.2, extraction of the correlation token set; analyze the attention scores of the output model for tokens through the attention attribution method, and extract the tokens with high attention scores to form the correlation token set;

[0073] Step 1.2.1, output of attention scores: Assume f is a trained model. Given a corpus D, for each input example e i , require the model to output each token in example e i : The corresponding attention scores where m is the number of tokens in the input;

[0074] Step 1.2.2, attention ranking: rank the attention scores for each example;

[0075] Step 1.2.3, extraction of correlation tokens: extract the top Top-K tokens in terms of attention ranking, and finally form a set of relevant tokens correlation set C = is the subset of correlation tokens related to the j-th label. Note: The relationship between the extracted top Top-K relevant tokens and the model prediction may be causal correlation or spurious correlation. These spurious correlations need to be forgotten by the model.

[0076] Step 1.3, neuron localization; by calculating the gradient of the proximity between the correlation token set and the causal token set or the spurious correlation token set with respect to the neuron, perform an integration operation to accurately locate the neurons related to spurious correlation information or causal correlation information, as Figure 2 shown;

[0077] Step 1.3.1, further define the correlation set: C is the correlation set obtained by the language model on the dataset D, denoted as:

[0078]

[0079] where, Denote the k-th neuron in the l-th layer of the model f, is value;

[0080] Step 1.3.2, the gradient integral of proximity is expressed as:

[0081]

[0082] Gradually change the value from 0 to the original value obtained from LM When α changes from 0 to 1, by integrating the gradient, accumulate the change in proximity caused by the change; if the neuron has a significant impact on proximity, then will be large, indicating that the neuron stores the corresponding causal or pseudo-correlated information.

[0083] Among them, denotes the gradient of proximity with respect to change; G ∈ (G cau , G spu ), if G = G cau , then this calculation formula is used to locate causal neurons, otherwise this formula is used to locate pseudo-correlated neurons;

[0084] denotes the proximity between

[0085]

[0086] Among them, denotes the number of tokens calculated in the intersection of two sets; is the sum of the attention scores of the tokens in

[0087] Step 1.3.3, integrate the above gradient to achieve the location of causal neurons and pseudo-correlated neurons.

[0088] Step 2, based on the particle swarm optimization algorithm, adjust the neuron parameters to adjust the model parameters to the optimal state; in this state, the model will forget the pseudo-correlation prior while striving to maintain the memory of causal correlation, as Figure 3 shown;

[0089] Step 2.1, according to different editing purposes and methods, classify the located neurons, which are neurons that only store pseudo-correlated information, neurons that only store causal-related information, and neurons with overlapping pseudo-correlated information and causal information;

[0090] Step 2.2: Design different editing methods based on different neuron categories. Specifically: for neurons that only store pseudo-correlated information, set their parameters to zero; for neurons that only store causally related information, keep their parameters unchanged; for neurons where pseudo-correlated information and causal information overlap, edit them using a method based on the PSO algorithm.

[0091] For neurons where pseudo-correlated information and causal information overlap, the specific method of editing them using a method based on the PSO algorithm is as follows:

[0092] Step 2.2.1: Particles and population: Define the number of neurons as n, and the set of neuron values as x = (x 1 , x 2 , …, x n ), where x n is the value of the nth neuron; in the optimization, regard this group of neurons as a particle, and multiple particles form a population X; each particle contains two pieces of information: position (x) and velocity (v). The position information is the parameter value of the neuron, and the velocity information describes the magnitude and direction of the change in the neuron value in the search space. These two pieces of information together determine the iterative value of the neuron;

[0093] Step 2.2.2: Evaluate the quality of the particle position through the fitness function fitness. The higher the fitness, the closer the particle position is to the optimal solution of the problem, expressed as:

[0094] fitness(x) = proximity(C (D,x) , G cau ) - proximity(C (D,x) , G spu )

[0095] where C (D,x) refers to the correlationset obtained from the corpus D when the parameters of the model neurons are x, proximity(C (D,x) , C cau ) is the degree of proximity between C and G cau , and Proximity(C (D,x) , G spu ) is the degree of proximity between C and G spu ;

[0096] Step 2.2.3: Iterative mechanism: The iterative mechanism combines individual learning and group learning; the particle updates its position according to its own optimal position p iUpdate its own position and velocity based on the information of the individual optimal position p and the global optimal position g; the individual optimal position is the position with the highest fitness found by a single particle during the search process, and the global optimal position is the position with the highest fitness found by the entire particle swarm during the search process; the mechanism is as follows:

[0097] v i,t+1 = wv i,t + c 1 r 1 (p i,t - x i,t ) + c 2 r 2 (g t - x i,t )

[0098] x i,t+1 = x i,t + v i,t+1

[0099] Among them, v i,t is the velocity of the i-th particle in the t-th iteration, w is the inertia weight, used to balance the influence of the particle's current velocity and past velocity; c 1 and c 2 are learning factors, used to adjust the learning speed; r 1 and r 2 are random numbers, used to introduce randomness; x i,t is the position of the i-th particle in the t-th iteration; by continuous iteration, the particle position x * that maximizes f can be obtained, that is, the optimal neuron parameter combination;

[0100] x * = argma x fitness(x)

[0101] Stopping criterion: Set the number of iterations to 50, and when this upper limit is reached, the optimization algorithm stops iterating; based on the above settings, optimize the neuron parameters and finally reach the optimal state.

[0102] A pseudo-correlation mitigation system based on machine forgetting according to the above method, as Figure 1 shown, the system includes: a neuron positioning module, a neuron editing module; the neuron positioning module is a neuron positioning module based on proximity, responsible for accurately identifying neurons related to pseudo-correlated information; the neuron editing module is responsible for editing the parameters of the located neurons to forget the pseudo-correlated prior of the model.

[0103] Performance evaluation on sentiment analysis and natural language inference tasks, where the average value can be used to evaluate the generalization ability of the model, and the evaluation results are shown in the following table:

[0104] Table 1: Performance (accuracy) of BERT models trained using different methods on the sentiment analysis dataset

[0105]

[0106] Table 2: Performance (accuracy) of BERT models trained using different methods on the natural language inference dataset

[0107]

[0108] Machine forgetting refers to the process by which a model partially or completely deletes the memory of specific data or information, and is commonly used in privacy protection research. Neuron parameter editing is an effective machine forgetting method that can fully and precisely delete target information without retraining the model. Inspired by this, we speculate that false correlation information may be stored in specific neurons, just like knowledge neurons. This hypothesis suggests that we can change the model's memory of pseudo-correlation information by locating and deleting (editing) these neurons (pseudo-correlation neurons).

[0109] Neuron localization. This module is responsible for accurately identifying the neurons related to pseudo-correlation information. The commonly used neuron localization method is the performance gradient response-based method, which locates neurons through the gradient response of the model's accuracy on a dataset without any false correlation features. However, such a dataset is difficult to construct.

[0110] Neuron parameter editing. The goal of this module is to edit the parameters of neurons to forget pseudo-correlated information. The commonly used method is to directly set the located neurons to zero. However, from the results of neuron localization, we find that there is an overlap between the neurons storing causal correlation information and the neurons storing false correlation information. Adjusting the parameters of neurons will change the model's memory of both causal relationships and false correlations, and the existing zero-setting method is no longer applicable.

[0111] The content not detailed in this invention specification belongs to the prior art well-known to those skilled in the art. Although the above describes the illustrative specific embodiments of the present invention for the convenience of those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

Claims

1. A method for alleviating pseudo-correlation based on machine forgetting, characterized in that: The following steps are involved: Step 1: locate pseudo-correlated neurons by changing the gradient response based on the proximity of neurons, where proximity is the closeness between the correlation token set extracted by the model and the causal token set or the pseudo-correlated token set; Step 1.1, construct a gold token set to guide the positioning of neurons; Including constructing causal marker sets and pseudo-correlation marker sets; Step 1.2, extraction of correlation token set; The attention score of the output model on the token is analyzed through the attention attribution method, and the tokens with high attention scores are extracted to form a correlation token set; Step 1.3, neuron positioning: by calculating the gradient of proximity with the change of neurons and performing integral operation, neurons with pseudo-correlated information or causal-related information can be accurately positioned; Step 2: Adjust the neuron parameters based on the particle swarm optimization algorithm to adjust the model parameters to the optimal state; Step 2.1, according to the purpose and method of editing, the located neurons are classified into neurons that only store pseudo-correlated information, neurons that only store causal-related information, and neurons that overlap pseudo-correlated information and causal information; Step 2.2, based on different neuron categories, design different editing methods, specifically: for neurons that will only store pseudo-correlated information, set their parameters to zero; for neurons that will only store causal-related information, keep their parameters unchanged; for neurons where pseudo-correlated information and causal information overlap, edit them using a method based on the PSO algorithm.

2. The method for alleviating pseudo-correlation based on machine forgetting according to claim 1, characterized in that: The step 1.1, constructing a gold token set to guide the positioning of neurons; The specific method of constructing causal tag sets and pseudo-correlation tag sets is: First, construct a causal tag set: Based on human prior knowledge of the training dataset, extract tokens that have a strong causal relationship with the label and construct a causal tag set represented as G cau ; Among them, label j is the jth label in this task; is with label j A set of tokens with causal relationships; Then, a pseudo-relevant tag set is constructed, including misclassified tokens and irrelevant tokens; Misclassified tokens, for the current label, misclassified tokens refer to tokens that have a causal relationship with other labels; using G cau Quickly get the misclassified token set label -j Indicates other labels except j label; is with label j A collection of related misclassified tokens; is with label -j A collection of related causal tokens; Irrelevant tokens: Based on human experience, we define tokens that are significantly irrelevant to the label as irrelevant tokens, and obtain a set of irrelevant tokens. The pseudo-correlation tag set is denoted as G spu ; 3. The method for alleviating pseudo-correlation based on machine forgetting according to claim 1, characterized in that: The step 1.2, extraction of correlation token set: analyzing the attention score of the output model to the token by the attention attribution method, and extracting the token with high attention score to form a correlation token set; specifically includes the following steps: Step 1.2.1, attention score output: Assuming f is a trained model, given a corpus D, for each input sample e i , requiring the model to output sample e i For each token in: The corresponding attention score Where m is the number of tokens in the input; Example e i The mth token in ; express The corresponding attention score; Step 1.2.2, attention sorting: sort the attention scores of each sample; Step 1.2.3, correlation token extraction: extract the top-K tokens of attention ranking, and finally form a correlation set on the entire corpus is the subset of correlation tokens associated with the jth tag.

4. The method for alleviating pseudo-correlation based on machine forgetting according to claim 1, characterized in that: The step 1.3, neuron positioning; by calculating the gradient of the proximity between the correlation token set and the causal tag set or the pseudo-correlation tag set as the neurons change, performing an integral operation, so as to accurately locate the neurons related to the pseudo-correlation information or the causal-correlation information, specifically includes the following steps: Step 1.3.1, further define the correlation set: C is the correlation set obtained by the language model on the corpus D, expressed as: in, represents the kth neuron in the lth layer of model f, yes The value of Step 1.3.2, the gradient integral of proximity is expressed as: Gradually The value of is changed from 0 to the original value obtained from LM When α changes from 0 to 1, by integrating the gradient, Accumulated due to Proximity changes caused by changes; in, Indicates proximity The gradient of change; G∈(G cau ,G spu ), if G=G cau , then the calculation formula is used to locate causal neurons, otherwise the formula is used to locate pseudo-correlation neurons; express The closeness between and G is expressed as: in, Indicates calculating the number of tokens in the intersection of two sets; yes The sum of the attention scores of the tokens in ; Step 1.3.3, integrate the above gradients to locate the causal neurons and pseudo-correlated neurons.

5. The method for alleviating pseudo-correlation based on machine forgetting according to claim 1, characterized in that: In step 2.2, for neurons with overlapping pseudo-correlated information and causal information, the PSO algorithm is used to edit them. The specific method is: Step 2.2.1, Particles and populations: Define the number of neurons as n, and the set of neuron values ​​as x = (x1, x2, ..., x n ), where x n is the value of the nth neuron; in the optimization, this group of neurons is regarded as a particle, and multiple particles form a population X; each particle contains two pieces of information: position and velocity. The position information is the parameter value of the neuron, and the velocity information describes the size and direction of the change of the neuron value in the search space. These two pieces of information together determine the iterative value of the neuron; Step 2.2.2, the fitness function fitness is used to evaluate the quality of the particle position. The higher the fitness, the closer the particle position is to the optimal solution of the problem, expressed as: fitness(x)=proximity(C (D,x) ,G cau )-proximity(C (D,x) ,G spu ) Among them, C (D,x) It refers to the correlation set obtained from the data set D when the parameter value of the model neuron is x, proximity (C (D,x) ,C cau ) is C and G cau The degree of proximity, proximity (C (D,x) ,G spu ) is C and G spu the degree of proximity; Step 2.2.3, iterative mechanism: The iterative mechanism adopts a combination of individual learning and group learning; particles are based on their own optimal position p i The information of the global optimal position g is used to update its own position and speed; the optimal position of a particle is the position with the highest fitness found by a single particle during the search process, and the global optimal position is the position with the highest fitness found by the entire particle group during the search process; the mechanism is as follows: v i,t+1 =wv i,t +c1r1(p i,t -x i,t )+c2r2(g t -x i,t ) x i,t+1 =x i,t +v i,t+1 Among them, v i,t is the speed of the i-th particle in the t-th iteration, w is the inertia weight, which is used to balance the influence of the current speed and the past speed of the particle; c1 and c2 are learning factors, which are used to adjust the learning speed; r1 and r2 are random numbers, which are used to introduce randomness; x i,t is the position of the i-th particle in the t-th iteration; through continuous iteration, the particle position x that maximizes fitness is obtained * , that is, the optimal neuron parameter combination; x * =argmax x fitness(x) Stop criteria: Set the number of iterations to 50. When this upper limit is reached, the optimization algorithm stops iterating. Based on the above settings, the parameters of the neurons are optimized and the optimal state is finally reached.

6. A pseudo-correlation mitigation system based on machine forgetting according to any one of claims 1 to 5, characterized in that: The system comprises: a neuron positioning module and a neuron editing module; The neuron localization module is a proximity-guided neuron localization module responsible for accurately identifying neurons related to pseudo-correlated information, where proximity is the degree of proximity between the correlation token set extracted by the model and the causal tag set or the pseudo-correlated tag set; The neuron editing module is responsible for editing the located neuron parameters to forget the pseudo-correlation prior of the model.

Images