Biomedical named entity recognition method based on sparse network

Abstract

The present invention discloses a sparse network-based biomedical named entity recognition method, comprising the steps of obtaining text data; constructing an initial model; training the initial model using data from all tasks to construct a shared sparse subnetwork and a private sparse subnetwork for each task; performing adversarial training to obtain a recognition model; and using the recognition model to recognize the current text to complete biomedical named entity recognition. The sparse network-based biomedical named entity recognition method provided by the present invention, through an innovative model construction and training process, not only achieves biomedical named entity recognition but also improves the utilization efficiency of model parameters, resulting in high reliability, good accuracy, and good results.

Description

Technical Field

[0001] The present invention belongs to the field of digital signal processing, and in particular relates to a biomedical named entity recognition method based on a sparse network. Background Art

[0002] The goal of biomedical named entity recognition (BioNER) is to identify various biomedical entities, such as DNA, genes, proteins, and diseases, from biomedical text. BioNER is a fundamental and crucial task, and many biomedical research areas rely on it, such as discovering gene function and extracting disease-therapy relationships. Therefore, building a BioNER system with high recognition accuracy is of great practical significance and application value.

[0003] Currently, traditional statistical learning-based methods rely heavily on manual feature engineering, which requires extensive domain knowledge (typically possessed only by domain experts) and results in poor system portability. Traditional deep learning-based methods, while not requiring manual feature engineering, do require large amounts of labeled data for model training. However, due to the difficulty and high cost of labeling, high-quality labeled data is extremely scarce, significantly impacting the effectiveness of deep learning methods. In recent years, some researchers have attempted to alleviate the lack of labeled data using multi-task learning. While these multi-task learning methods have achieved promising results, they still face challenges: For example, they suffer from low parameter efficiency and negative interactions between tasks, sometimes even leading to decreased model performance. Summary of the Invention

[0004] The purpose of the present invention is to provide a biomedical named entity recognition method based on a sparse network with high reliability, good accuracy and good effect, which can improve the utilization efficiency of model parameters and alleviate the problem of lack of labeled data for a single task through training data of multiple tasks.

[0005] The present invention provides a biomedical named entity recognition method based on a sparse network, comprising the following steps:

[0006] S1. Obtain text data;

[0007] S2. Build the initial model;

[0008] S3. Train the initial model constructed in step S2 using data from all tasks to construct a shared sparse subnetwork and a private sparse subnetwork for each task.

[0009] S4. Perform adversarial training to obtain a recognition model;

[0010] S5. Use the recognition model obtained in step S4 to recognize the current text, thereby completing biomedical named entity recognition.

[0011] The construction of the initial model described in step S2 specifically includes the following steps:

[0012] The initial model includes a word embedding layer, a character embedding layer, a character-level bidirectional long short-term memory network, a word-level bidirectional long short-term memory network, and a conditional random field;

[0013] θ is the parameter of the encoder part in the initial model; there are T tasks in total, and each task is numbered t. For all the training data of T tasks, D t is the dataset for task t, N t is the number of training samples for task t, S k is the word sequence of the kth training sample and S k ={w1,w2,...,w n},w i is the i-th word, Y k is the label sequence of the kth training sample and Y k ={y1,y2,...,y n},y i is the label corresponding to the i-th word; M s is the mask matrix of the shared sparse sub-network, M t is the mask matrix of the private sparse subnetwork of task t.

[0014] Step S3 uses the data of all tasks to train the initial model constructed in step S2, thereby constructing a shared sparse subnetwork and a private sparse subnetwork for each task, specifically including the following steps:

[0015] A. Randomly initialize the entire neural network, including initializing the cropping rate pr, initializing the cropping interval itv, and initializing the target sparsity of the shared sparse subnetwork;

[0016] B. Initialize the mask matrix M of the shared sparse sub-network s : The mask matrix M of the shared sparse sub-network s All elements in are initialized to 1, represents the parameters in the shared sparse sub-network, θ is the parameter of the encoder part in the initial model, is the Hadamard product of the matrix; step is the number of training steps and is initialized to step = 0;

[0017] C. Randomly select a batch of training samples B from a dataset of a task t ;

[0018] D. Use the encoder to encode the input text: including using the word embedding layer to convert the words in the text sequence into word embedding vectors, and using the character embedding layer to convert the characters in the text sequence into character embedding vectors; using the shared sparse subnetwork in the character-level bidirectional long short-term memory network to encode the character embedding of the text to obtain the character-level representation of the word; concatenating the word embedding and character-level representation of each word and inputting them into the word-level bidirectional long short-term memory network; using the shared sparse subnetwork in the word-level bidirectional long short-term memory network to encode and obtain the context representation R of each word. k R k ={r1,r2,...,r n}, r i For word w i The context of step D is represented as

[0019] E. The sequence R obtained after encoding by the encoder k and Y k Input into the conditional random field of task t and calculate the conditional probability Where n is the tag sequence Y k length, Y' is a possible label sequence for the kth training sample, φ(R k ) is all possible label sequences of the kth training sample, ψ(y i-1 ,y i ,r i ) is the scoring function of the label sequence and For the label pair (y i-1 ,y i ), For the label pair (y i-1 ,y i ) bias;

[0020] F. Calculate the loss loss as is a batch of training samples B t The label sequence in m is the training sample batch B t The number of training samples in ;

[0021] G. Perform back propagation to obtain the gradient of each parameter;

[0022] H. Update all parameters using gradient descent algorithm;

[0023] I. The number of training steps increases by 1;

[0024] If the value of the training step number step is an integer multiple of the trimming interval itv, then The parameters in are sorted by absolute value, and the parameters with smaller absolute value pr% are cut off; the parameters with smaller absolute value pr% are cut off, specifically, M s The corresponding elements in are set to 0;

[0025] J. Repeat steps C to I until the shared sparse subnetwork is Sparsity Less than the target sparsity; where ||M s ||0 is M s The L0 norm of M s is the mask matrix of the shared sparse subnetwork, |θ| is the number of parameters in θ;

[0026] K. Settings M free is the mask matrix of the remaining parameters after removing the shared sparse subnetwork, is the mask matrix M for the shared sparse sub-network s Perform inversion, that is, invert the elements that were originally 0 in the mask matrix to 1, and invert the elements that were originally 1 to 0;

[0027] L. Randomly set unused parameters in Parameters are assigned to task t, thereby building a private sparse subnetwork for task t, and the corresponding mask matrix is M t ;|D t | is the number of training samples for task t, |D| is the total number of training samples for all tasks;

[0028] M.Update M free The value of M free ' is the M before the update free The value of

[0029] N. Repeat steps L to M until all tasks t are traversed;

[0030] O. Output the mask matrix M of the shared sparse sub-network s , and the mask matrix M of the private sparse sub-network of each task t .

[0031] The adversarial training described in step S4 to obtain a recognition model specifically includes the following steps:

[0032] a. Set the number of training steps to 0; set the adversarial training interval itv and the maximum number of training rounds max_epoch;

[0033] b. If the number of training steps step is an integer multiple of the adversarial training interval itv, set the adversarial loss adv_loss to 0 and execute steps c to i. Otherwise, go directly to step j.

[0034] c. From the dataset D of task t t Extract a batch of training data B from t Conduct training; B t is a batch of training data extracted from task t;

[0035] d. Use a shared sparse subnetwork to generate task-independent shared features for sentence S: in r i For word w i Contextual representation of ;

[0036] e.Yes The average is performed to obtain the vector representation r of the sentence; then, the task discriminator predicts the task p to which the sentence belongs k =softmax(Wr+b), where p k is the probability that the sentence of the kth sample predicted by the discriminator belongs to each task, softmax() is the normalized exponential function, W is the weight in the discriminator, and b is the bias in the discriminator;

[0037] f. Calculate loss adv (L,P) is where l k t is the label of the kth training sample in adversarial training. If the kth training sample comes from task t, then l k t is 1, l k t Otherwise, it is 0. k t is the probability that the kth training sample predicted by the discriminator belongs to task t, and m is the training sample batch B t The number of samples included in ;

[0038] g. Update the adversarial loss adv_loss to adv_loss'+loss adv (L, P), where adv_loss' is the value of adv_loss before the update;

[0039] h. Repeat steps c to g until all tasks t are traversed;

[0040] i. Based on the value of adv_loss, the discriminator is updated using the gradient descent algorithm; then, based on the value of adv_loss, the generator is updated using the gradient ascent algorithm

[0041] j. Randomly sample a batch of training data B from the training set D t ;

[0042] k. Use shared sparse subnetworks and private sparse subnetworks for sentence S k coding, Among them S k ={w1,w2,...,w n} is the training batch B t The sentence of the kth training sample in w i is the i-th word in the sentence, R k ={r1,r2,...,r n} is a sentence S k The context representation after encoding by the shared sparse sub-network and the private sparse sub-network, r i is the context representation of the i-th word, M t is the mask matrix of the private sparse subnetwork of task t, M s is the mask matrix of the shared sparse sub-network;

[0043] l. Calculate losses is a batch of training samples B t The label sequence in m is the training sample B t The number of training samples in ;

[0044] m. Update the sub-network parameters and other parameters through the gradient descent algorithm according to the loss ner_loss; the parameters of the sub-network are Including parameters of shared sparse sub-network and parameters of private sparse sub-network;

[0045] n. The value of the training step step increases by 1;

[0046] o. Repeat steps b to n until all data in the training data D are trained;

[0047] Repeat steps b to o until the number of training rounds reaches the set maximum number of training rounds max_epoch; obtain the final recognition model.

[0048] The sparse network-based biomedical named entity recognition method provided by the present invention not only realizes the recognition of biomedical named entities through an innovative model construction and training process, but also improves the utilization efficiency of model parameters and alleviates the problem of lack of labeled data for a single task through training data of multiple tasks. It has high reliability, good accuracy and good effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0050] like Figure 1 The figure shows a flow chart of the method of the present invention: The method of biomedical named entity recognition based on sparse network provided by the present invention comprises the following steps:

[0051] S1. Obtain text data;

[0052] S2. Build an initial model; specifically, the following steps are included:

[0053] The initial model includes a word embedding layer, a character embedding layer, a character-level bidirectional long short-term memory network, a word-level bidirectional long short-term memory network, and a conditional random field;

[0054] θ is the parameter of the encoder part in the initial model; there are T tasks in total, and each task is numbered t. For all the training data of T tasks, D t is the dataset for task t, N t is the number of training samples for task t, S k is the word sequence of the kth training sample and S k ={w1,w2,...,w n},w i is the i-th word, Y k is the label sequence of the kth training sample and Y k ={y1,y2,...,y n},y i is the label corresponding to the i-th word; M s is the mask matrix of the shared sparse sub-network, M t is the mask matrix of the private sparse subnetwork of task t.

[0055] S3. Train the initial model constructed in step S2 using data from all tasks to build a shared sparse subnetwork and a private sparse subnetwork for each task. This includes the following steps:

[0056] A. Randomly initialize the entire neural network, including initializing the cropping rate pr, initializing the cropping interval itv, and initializing the target sparsity of the shared sparse subnetwork;

[0057] B. Initialize the mask matrix M of the shared sparse sub-network s : The mask matrix M of the shared sparse sub-network s All elements in are initialized to 1, represents the parameters in the shared sparse sub-network, θ is the parameter of the encoder part in the initial model, is the Hadamard product of the matrix; step is the number of training steps, and is initialized to step = 0;

[0058] C. Randomly select a batch of training samples B from a dataset of a task t ;

[0059] D. Use the encoder to encode the input text: including using the word embedding layer to convert the words in the text sequence into word embedding vectors, and using the character embedding layer to convert the characters in the text sequence into character embedding vectors; using the shared sparse subnetwork in the character-level bidirectional long short-term memory network to encode the character embedding of the text to obtain the character-level representation of the word; concatenating the word embedding and character-level representation of each word and inputting them into the word-level bidirectional long short-term memory network; using the shared sparse subnetwork in the word-level bidirectional long short-term memory network to encode and obtain the context representation R of each word. k R k ={r1,r2,...,r n}, r i For word w i The context of step D is represented as

[0060] E. The sequence R obtained after encoding by the encoder k and Y k Input into the conditional random field of task t and calculate the conditional probability p(Y k |R k )for Where n is the tag sequence Y k length, Y' is a possible label sequence for the kth training sample, φ(R k ) is all possible label sequences of the kth training sample, ψ(y i-1 ,y i ,r i ) is the scoring function of the label sequence and For the label pair (y i-1 ,y i ), For the label pair (y i-1 ,y i ) bias;

[0061] F. Calculate the loss loss as is a batch of training samples B t The label sequence in m is the training sample batch B t The number of training samples in ;

[0062] G. Perform back propagation to obtain the gradient of each parameter;

[0063] H. Update all parameters using gradient descent algorithm;

[0064] I. The number of training steps increases by 1;

[0065] If the value of the training step number step is an integer multiple of the trimming interval itv, then The parameters in are sorted by absolute value, and the parameters with smaller absolute value pr% are cut off; the parameters with smaller absolute value pr% are cut off, specifically, M s The corresponding elements in are set to 0;

[0066] J. Repeat steps C to I until the shared sparse subnetwork is Sparsity Less than the target sparsity; where ||M s ||0 is M s The L0 norm of M s is the mask matrix of the shared sparse subnetwork, |θ| is the number of parameters in θ;

[0067] K. Settings M free is the mask matrix of the remaining parameters after removing the shared sparse subnetwork, is the mask matrix M for the shared sparse sub-network s Perform inversion, that is, invert the elements that were originally 0 in the mask matrix to 1, and invert the elements that were originally 1 to 0;

[0068] L. Randomly set unused parameters in Parameters are assigned to task t, thereby building a private sparse subnetwork for task t, and the corresponding mask matrix is M t ;|D t | is the number of training samples for task t, |D| is the total number of training samples for all tasks;

[0069] M.Update M free The value of M free ' is the M before the updatefree The value of

[0070] N. Repeat steps L to M until all tasks t are traversed;

[0071] O. Output the mask matrix M of the shared sparse sub-network s , and the mask matrix M of the private sparse sub-network of each task t .

[0072] S4. Perform adversarial training to obtain a recognition model; specifically, the steps include:

[0073] a. Set the number of training steps to 0; set the adversarial training interval itv and the maximum number of training rounds max_epoch;

[0074] b. If the number of training steps step is an integer multiple of the adversarial training interval itv, set the adversarial loss adv_loss to 0 and execute steps c to i. Otherwise, go directly to step j.

[0075] c. From the dataset D of task t t Extract a batch of training data B from t Conduct training; B t is a batch of training data extracted from task t;

[0076] d. Use a shared sparse subnetwork to generate task-independent shared features for sentence S: in r i For word w i Contextual representation of ;

[0077] e.Yes The average is performed to obtain the vector representation r of the sentence; then, the task discriminator predicts the task p to which the sentence belongs k =softmax(Wr+b), where p k is the probability that the sentence of the kth sample predicted by the discriminator belongs to each task, softmax() is the normalized exponential function, W is the weight in the discriminator, and b is the bias in the discriminator;

[0078] f. Calculate loss adv (L,P) is where l k t is the label of the kth training sample in adversarial training. If the kth training sample comes from task t, then l k t is 1, l k t Otherwise, it is 0. kt is the probability that the kth training sample predicted by the discriminator belongs to task t, and m is the training sample batch B t The number of samples included in ;

[0079] g. Update the adversarial loss adv_loss to adv_loss'+loss adv (L, P), where adv_loss' is the value of adv_loss before the update;

[0080] h. Repeat steps c to g until all tasks t are traversed;

[0081] i. Based on the value of adv_loss, the discriminator is updated using the gradient descent algorithm; then, based on the value of adv_loss, the generator is updated using the gradient ascent algorithm

[0082] j. Randomly sample a batch of training data B from the training set D t ;

[0083] k. Use shared sparse subnetworks and private sparse subnetworks for sentence S k coding, Among them S k ={w1,w2,...,w n} is the training batch B t The sentence of the kth training sample in w i is the i-th word in the sentence, R k ={r1,r2,...,r n} is a sentence S k The context representation after encoding by the shared sparse sub-network and the private sparse sub-network, r i is the context representation of the i-th word, M t is the mask matrix of the private sparse subnetwork of task t, M s is the mask matrix of the shared sparse sub-network;

[0084] l. Calculate the loss ner_loss as is a batch of training samples B t The label sequence in m is the training sample B t The number of training samples in ;

[0085] m. Update the sub-network parameters and other parameters through the gradient descent algorithm according to the loss ner_loss; the parameters of the sub-network are Including parameters of shared sparse sub-network and parameters of private sparse sub-network;

[0086] n. The value of the training step step increases by 1;

[0087] o. Repeat steps b to n until all data in the training data D are trained;

[0088] Repeat steps b to o until the number of training rounds reaches the set maximum number of training rounds max_epoch; obtain the final recognition model.

[0089] S5. Use the recognition model obtained in step S4 to recognize the current text, thereby completing biomedical named entity recognition.

[0090] The advantages of the method of the present invention are described below with reference to an embodiment:

[0091] Table 1 shows a comparative data table of the method of the present invention and other methods under the experimental setting of 5 data sets.

[0092] Table 1 Comparative data diagram

[0093]

[0094] The experimental results, using the F1 value as the evaluation metric, were obtained from three independent rounds of experiments. The mean and standard deviation of the F1 value for our method are listed. The experimental results show that our method outperforms the comparison methods on all four datasets.

[0095] In addition, each method compared in Table 1 uses 100% of the parameters of the entire model. Although the method of the present invention achieves good results, it does not use 100% of the model parameters for each task. This shows that the method of the present invention can greatly improve the efficiency of model parameters.

[0096] In the method of the present invention, only one shared sparse subnetwork and one private sparse subnetwork are used for each task. Table 2 shows the sparsity of each sparse subnetwork in the method of the present invention, that is, the percentage of the number of parameters in the sparse subnetwork to the number of parameters in the character-level BiLSTM (bidirectional long short-term memory network) and word-level BiLSTM (bidirectional long short-term memory network).

[0097] Table 2 Schematic diagram of the proportion of parameters in sparse sub-networks

[0098]

[0099] It can be seen that in the method of the present invention, on average only 44% of the parameters in Character-level BiLSTM and Word-level BiLSTM are used for each task.

Claims

1. A biomedical named entity recognition method based on a sparse network, comprising the following steps: S1. Obtain text data; S2. Build the initial model; S3. Train the initial model constructed in step S2 using data from all tasks to build a shared sparse subnetwork and a private sparse subnetwork for each task. This includes the following steps: A. Randomly initialize the entire neural network, including initializing the cropping rate pr, initializing the cropping interval itv, and initializing the target sparsity of the shared sparse subnetwork; B. Initialize the mask matrix M of the shared sparse sub-network s : The mask matrix M of the shared sparse sub-network s All elements in are initialized to 1, represents the parameters in the shared sparse sub-network, θ is the parameter of the encoder part in the initial model, is the Hadamard product of the matrix; step is the number of training steps, and is initialized to step = 0; C. Randomly select a batch of training samples B from a dataset of a task t ; D. Encode the input text using an encoder; E. The sequence R obtained after encoding by the encoder k and Y k Input into the conditional random field of task t and calculate the conditional probability p(Y k |R k );in, Y k is the label sequence of the kth training sample, R k is the context representation of the word; F. Calculate the loss loss as is a batch of training samples B t The label sequence in m is the training sample batch B t The number of training samples in ; G. Perform back propagation to obtain the gradient of each parameter; H. Update all parameters using gradient descent algorithm; I. The number of training steps increases by 1; If the value of the training step number step is an integer multiple of the trimming interval itv, then The parameters in are sorted by absolute value, and the parameters with smaller absolute value pr% are cut off; the parameters with smaller absolute value pr% are cut off, specifically, M s The corresponding elements in are set to 0; J. Repeat steps C to I until the shared sparse subnetwork is Sparsity Less than the target sparsity; where ||M s ||0 is M s The L0 norm of M s is the mask matrix of the shared sparse subnetwork, |θ| is the number of parameters in θ; K. Settings M free is the mask matrix of the remaining parameters after removing the shared sparse subnetwork, is the mask matrix M for the shared sparse sub-network s Perform inversion, inverting the elements in the mask matrix that were originally 0 to 1, and inverting the elements that were originally 1 to 0; L. Randomly set unused parameters in Parameters are assigned to task t, thereby building a private sparse subnetwork for task t, and the corresponding mask matrix is M t ;|D t | is the number of training samples for task t, |D| is the total number of training samples for all tasks; M.Update M free The value of M free ' is the M before the update free The value of N. Repeat steps L to M until all tasks t are traversed; O. Output the mask matrix M of the shared sparse sub-network s , and the mask matrix M of the private sparse sub-network of each task t ; S4. Perform adversarial training to obtain a recognition model; S5. Use the recognition model obtained in step S4 to recognize the current text, thereby completing biomedical named entity recognition.

2. The biomedical named entity recognition method based on sparse network according to claim 1 is characterized in that The construction of the initial model described in step S2 specifically includes the following steps: The initial model includes a word embedding layer, a character embedding layer, a character-level bidirectional long short-term memory network, a word-level bidirectional long short-term memory network, and a conditional random field; θ is the parameter of the encoder part in the initial model; there are T tasks in total, and each task is numbered t. For all the training data of T tasks, D t is the dataset for task t, N t is the number of training samples for task t, S k is the word sequence of the kth training sample and S k ={w1,w2,...,w n },w i is the i-th word, Y k is the label sequence of the kth training sample and Y k ={y1,y2,...,y n },y i is the label corresponding to the i-th word.

3. The biomedical named entity recognition method based on sparse network according to claim 2 is characterized in that Encoding the input text using an encoder as described in step D specifically includes the following steps: The word embedding layer is used to convert the words in the text sequence into word embedding vectors, and the character embedding layer is used to convert the characters in the text sequence into character embedding vectors; the shared sparse subnetwork in the character-level bidirectional long short-term memory network is used to encode the character embedding of the text to obtain the character-level representation of the word; the word embedding and character-level representation of each word are spliced and input into the word-level bidirectional long short-term memory network; the shared sparse subnetwork in the word-level bidirectional long short-term memory network is used to encode and obtain the context representation R of each word. k R k ={r1,r2,...,r n }, r i For word w i The context of step D is represented as The sequence R obtained after being encoded by the encoder in step E k and Y k Input into the conditional random field of task t and calculate the conditional probability p(Y k |R k ), specifically including the following steps: Where n is the tag sequence Y k length, Y' is a possible label sequence for the kth training sample, φ(R k ) is all possible label sequences of the kth training sample, ψ(y i-1 ,y i ,r i ) is the scoring function of the label sequence and For the label pair (y i-1 ,y i ), For the label pair (y i-1 ,y i ) bias.

4. The biomedical named entity recognition method based on sparse network according to claim 3 is characterized in that The adversarial training described in step S4 to obtain a recognition model specifically includes the following steps: a. Set the number of training steps to 0; set the adversarial training interval itv and the maximum number of training rounds max_epoch; b. If the number of training steps step is an integer multiple of the adversarial training interval itv, set the adversarial loss adv_loss to 0 and execute steps c to i. Otherwise, go directly to step j. c. From the dataset D of task t t Extract a batch of training data B from t Conduct training; B t is a batch of training data extracted from task t; d. Use a shared sparse subnetwork to generate task-independent shared features for sentence S: in r i For word w i Contextual representation of ; e.Yes The average is performed to obtain the vector representation r of the sentence; then, the task discriminator predicts the task p to which the sentence belongs k =softmax(Wr+b), where p k is the probability that the sentence of the kth sample predicted by the discriminator belongs to each task, softmax() is the normalized exponential function, W is the weight in the discriminator, and b is the bias in the discriminator; f. Calculate loss adv (L,P) is where l k t is the label of the kth training sample in adversarial training. If the kth training sample comes from task t, then l k t is 1, l k t Otherwise, it is 0. k t is the probability that the kth training sample predicted by the discriminator belongs to task t, and m is the training sample batch B t The number of samples included in ; g. Update the adversarial loss adv_loss to adv_loss'+loss adv (L, P), where adv_loss' is the value of adv_loss before the update; h. Repeat steps c to g until all tasks t are traversed; i. Based on the value of adv_loss, the discriminator is updated using the gradient descent algorithm; then, based on the value of adv_loss, the generator is updated using the gradient ascent algorithm j. Randomly sample a batch of training data B from the training set D t ; k. Use shared sparse subnetworks and private sparse subnetworks for sentence S k coding, Among them S k ={w1,w2,...,w n } is the training batch B t The sentence of the kth training sample in w i is the i-th word in the sentence, R k ={r1,r2,...,r n } is a sentence S k The context representation after encoding by the shared sparse sub-network and the private sparse sub-network, r i is the context representation of the i-th word, M t is the mask matrix of the private sparse subnetwork of task t, M s is the mask matrix of the shared sparse sub-network; l. Calculate the loss ner_loss as is a batch of training samples B t The label sequence in m is the training sample B t The number of training samples in ; m. Update the sub-network parameters and other parameters through the gradient descent algorithm according to the loss ner_loss; the parameters of the sub-network are Including parameters of shared sparse sub-network and parameters of private sparse sub-network; n. The value of the training step step increases by 1; o. Repeat steps b to n until all data in the training data D are trained; Repeat steps b to o until the number of training rounds reaches the set maximum number of training rounds max_epoch; obtain the final recognition model.

Images