Named entity identification method fusing tuning mode

Abstract

The invention discloses a named entity recognition method fusing an adjustment and optimization mode, and relates to the technical field of named entity recognition, and the method comprises the steps: using a boundary detection model to recognize a starting boundary and an ending boundary of a candidate entity, obtaining a set of all candidate entities, setting two labels to be inserted into the two sides of the candidate entity, and obtaining a sentence marked with the candidate entity; forming a prompt template by using the obtained candidate entities, filling an input sequence by using the prompt template and sentences marked with the candidate entities, and obtaining word representation through a pre-training language model; based on a gating mechanism, a deep learning network is used for training, semantics of different dimensions are extracted, including global semantics and mask semantics, and then the named entity is predicted. Therefore, by the adoption of the named entity recognition method fusing the tuning mode, various semantic representations can be obtained through the pre-training language model, the representations of the candidate entities are enriched, and interference of other potential candidate entities is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of named entity recognition, and in particular to a named entity recognition method integrating a tuning approach. Background Art

[0002] Named Entity Recognition (NER) aims to identify text spans that belong to predefined categories, such as names of people, organizations, addresses, etc. Named entity recognition is a fundamental task in natural language processing, and its results can promote the research of downstream tasks such as relationship extraction, question-answering systems, and sentiment analysis. Named entities in sentences generally exist in two forms: flat and nested. Nested named entities refer to situations where an entity contains one or more other entities. The emphasis of identifying flat entities and nested entities is different. Since each word in a sentence may have multiple tags, named entities with nested structures significantly affect the overall named entity recognition performance and even bring obstacles to many natural language processing tasks.

[0003] Currently, existing models are all built on pre-trained language models (PLMs), such as BERT, which provide rich semantics for NER tasks. Pre-trained language models utilize massive amounts of unlabeled corpus data to learn universal semantic representations, containing a wealth of knowledge that is helpful for NER tasks. The paradigms of this model can be mainly divided into two categories: fine-tuning and prompt tuning. In the fine-tuning method, PLMs are used to convert words into high-dimensional abstract representations; based on the dense vector representation, the probability of a specified type is output; finally, the neural network is optimized using a task-specific objective function. In prompt tuning, the downstream task is customized to a language modeling task, and a cloze test is constructed as a prompt to guide the model to stimulate the potential of the PLM. The advantage of prompt tuning is that the downstream task goal is customized to the goal of the PLM. The decision-making process is the same as the training process of the PLM, and different tuning methods have different incentives for the model.

[0004] However, hint learning methods only combine templates and instances on the input surface, lacking a deep semantic interaction structure, which limits the expressive power of hint guidance. Fine-tuning focuses on learning contextual features from training data, lacking semantic motivation at the task level, and thus tends to overlook the semantic objectives of the task itself. Given that a sentence often contains multiple candidate entities with similar contexts, some of which are nested, existing methods are susceptible to interference from other potential candidates when identifying the current candidate.

[0005] To this end, the present invention provides a selective and adjustable fusion mechanism to achieve on-demand regulation of semantic contribution, enrich the multi-dimensional semantic representation of candidate entities, reduce the interference of candidate entities on the current entity recognition task, and provide a reference for the recognition of nested entities. Summary of the Invention

[0006] The purpose of the present invention is to provide a named entity recognition method that integrates tuning, which can make full use of hint semantics and fine-tuning semantics to solve the problem that the existing technology is interfered with by other potential candidate entities when identifying the current candidate entity.

[0007] To achieve the above object, the present invention provides a method for named entity recognition using a fusion tuning approach, comprising the following steps:

[0008] S1. Use the boundary detection model to identify the starting and ending boundaries of the candidate entity, obtain the set of all candidate entities, and set two labels to insert on both sides of the candidate entity to obtain the sentence marked with the candidate entity;

[0009] S2. The obtained candidate entities are combined into a prompt template, and the prompt template and the sentences marked with the candidate entities are used to fill the input sequence, and the word representation is obtained through the pre-trained language model;

[0010] S3. Based on the gating mechanism, deep learning networks are used for training to extract semantics of different dimensions, including global semantics and mask semantics, and then predict named entities.

[0011] Preferably, step S2 comprises the following steps:

[0012] S21. After the input sequence passes through the word embedding layer and encoding layer of the pre-trained language model, the input sequence semantics is obtained, and the classifier is used to obtain the probabilities of different entity types, as follows:

[0013] H = BERT(T);

[0014] R=Classifier(H [CLS] );

[0015] Where T is the input sequence, H is the sequence semantics, R is the probability of different entity types, Classifier() is the classifier, and H [CLS] The output vector of the sentence classification head CLS;

[0016] S22, the self-attention mechanism, and BiLSTM form an interaction layer to convert the sequence semantics H into word representations as follows:

[0017] α=Softmax((H·W q )(H.W k ) T );

[0018] Z=α·(H·W v );

[0019] Z′=BiLSTM(Z);

[0020] Where Z′ is the sequence vector representation, α is the similarity between the query and the key, Z is the predicted sequence, and W q 、W k 、W v is the weight matrix of the self-attention mechanism.

[0021] Preferably, in step S3, the deep learning network includes a first gated memory unit and a second gated memory unit; wherein the first gated memory unit filters information of different entity types to extract fine-tuned semantics; and the second gated memory unit retains the global semantics required by the mask to extract the prompt semantics.

[0022] Preferably, the first gated memory unit obtains the masked semantics of the sentence by filtering the global semantics and the mask vector, as follows:

[0023] Z″ [MASK] =MLP(Z′ [MASK] );

[0024] c=σ(W c (R+Z″ [MASK] )+b c );

[0025] C keep =(1-c)·Z″ [MASK] ;

[0026] C next =c·Z″ [MASK] ;

[0027] Where W c 、b c are the weight matrix and bias of the first gate control unit respectively, and c is the update gate; C keep Represented as the first mask semantics as part of the prediction result; C next is the second mask semantics, serving as the input of the second gated memory unit.

[0028] Preferably, the second gated memory unit determines the retained global semantics according to the second mask semantics as follows:

[0029] p=σ(W m (C next +R)+b m );

[0030] P choose =p·R;

[0031] In the formula, p represents the weight ratio of global semantics, P chooserepresents mask semantics, R represents global semantics, and σ is the activation function.

[0032] Preferably, in step S3, the predicted named entity is represented as:

[0033] P=C keep +P choose +Z″ [MASK] ;

[0034] Where P is the predicted entity type.

[0035] Preferably, in S3, predicting the named entity includes calculating the predicted value of each candidate instance using a gated memory mechanism and performing a cross entropy operation with the true value to obtain the training loss, as follows:

[0036]

[0037] Where Loss is the training loss function, n is the number of entities, and y i is the true value of the i-th entity, P i is the predicted value of the i-th entity.

[0038] Therefore, the present invention adopts the above-mentioned named entity recognition method of a fusion tuning method, which has the following technical effects:

[0039] The present invention fully utilizes global semantics and mask semantic information to make decisions by fusing the semantics obtained by extraction fine-tuning and prompt tuning; at the same time, it uses a gated fusion module to reconstruct various types of semantic information into a multidimensional semantic representation, and uses a deep learning network to train and optimize related target tasks.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The diagram is a network architecture diagram of an embodiment of a named entity recognition method in a fusion tuning manner. DETAILED DESCRIPTION

[0042] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0043] The input of existing cue learning methods consists of instances and cue templates. Since the semantics of instances and cues differ, instance semantics are objective information about the instance content itself, while cue semantics are task-oriented. The former focuses on global content understanding, while the latter focuses on task guidance, requiring the interaction between global semantics and local information. Existing cue learning methods only splice templates and instances on the input surface, lacking a deep semantic interaction structure, which limits the expressiveness of cue guidance. Fine-tuning, on the other hand, focuses on learning contextual features from training data, lacks semantic motivation at the task level, and therefore tends to overlook the semantic objectives of the task itself. As can be seen, existing cue learning methods lack a dynamic semantic association mechanism between the global context and mask of the template and instance, which in turn limits the interaction between global and local semantics and makes the current recognition result susceptible to interference from other candidate entities. Therefore, designing a selective and adjustable fusion mechanism to achieve on-demand regulation of semantic contribution is a key challenge.

[0044] To address the above issues, the present invention provides a named entity recognition method based on fusion tuning, which combines fine-tuning and prompt tuning, optimizes the target of fine-tuning and pre-training tasks, and then introduces a gated fusion mechanism to dynamically integrate information from different semantic sources, significantly enriching the semantic representation of each candidate entity. Figure 1 The advantage of this method is that it stimulates the potential of pre-training in multiple dimensions, combines semantics with different tuning, and obtains multi-semantic representations. It uses a gated fusion module to fuse the fine-tuned representation and the hinted representation. The special fusion module consists of two gating mechanisms, one for extracting the fine-tuned semantics and the other for extracting the hinted semantics.

[0045] The above method mainly includes three parts, as follows:

[0046] In the first part, the start and end positions of candidate entities are predicted through the boundary detection task and combined into candidate entities. For each candidate entity, labels are injected on both sides and the candidate entity is classified as follows:

[0047] (1) Candidate entity selection:

[0048] S=[x1,x2,...,x n ] represents a sentence of length n. i Represents the i-th word in S. E ij Represents the candidate entity with starting position i and ending position j. All candidate entities in the sentence can be represented as E={E ij |1≤i≤j≤n}.

[0049] In this embodiment, a boundary detection model, such as the bert-attention-bilstm model, is used to identify the start and end boundaries of all candidate entities in the sentence, denoted as B and E respectively, to generate a candidate entity set, which can effectively reduce the number of invalid candidate entities. For all candidate entities that meet E ij ∈E,x i ∈B and x j ∈E.

[0050] (2) For each candidate entity E ij ∈E, set two labels (t b =[B], t e =[E]) are inserted on both sides of the candidate entity. The sentence is reformulated as:

[0051] S ij =[x1,...,x i-1 ,t b ,...,t e ,x j+1 ,...,x n ];

[0052] Where subscripts i and j represent the labeled candidate entity locations.

[0053] The second part uses pre-trained language models (PLMs) as encoders to obtain word representations. In this part, the model input includes a prompt template and a sentence labeled with candidate entities (such as "is an [MASK]entity"). The prompt template M consists of generated candidate entities and is used to convert the raw input into a sequence representation with task semantics to stimulate the potential of the pre-trained language model. The specific operation process is as follows:

[0054] First, the input sequence T consists of sentences S labeled with candidate entities ij And the prompt template M is filled, the formula is:

[0055] T=[x1,...,x i-1 ,t b ,...,t e ,x j+1 ,x n .[SEP],x i ,...,x j ,is,an,[MASK],entity];

[0056] After the word embedding layer and encoding layer of the pre-trained language model, the input sequence semantics H is obtained:

[0057] H = BERT(T).

[0058] Secondly, the last layer output vector H of the sentence classification head [CLS] is used [CLS] , a classifier is used to obtain the probability R of different entity types, which is used to represent the global semantics and serve as the input of the subsequent gated fusion module.

[0059] R=Classifier(H [CLS] );

[0060] Among them, Classifier() is a classifier, which is a linear layer.

[0061] Then, through the interaction layer consisting of the self-attention mechanism and BiLSTM, the sequence semantics H is converted into a sequence vector representation Z′ as follows:

[0062] α=Softmax((H·W q )(H.W k ) T );

[0063] Z=α·(H·W v );

[0064] Z′=BiLSTM(Z);

[0065] Where W q 、W k 、W v is the weight matrix of the self-attention mechanism, and Z is the prediction sequence.

[0066] In the third part, a deep network is used to obtain semantics of different dimensions, and a gated fusion module (GFM) is used to fuse hint tuning and fine-tuning to obtain a multi-dimensional representation. This method can dynamically integrate information from different semantic sources, significantly enriching the semantic representation of each candidate entity.

[0067] Specifically, after the pre-trained language model in the second part, the input special word segmentation [CLS] has global semantics in the output, which is often used in classification problems. To extract multi-dimensional semantics, this embodiment introduces a gating mechanism that combines fine-tuning and prompt information. The prompt information focuses on stimulating the model's potential at the task level, while the fine-tuning information focuses on capturing the global semantic representation of the input sample at the data level, improving the model's ability to understand entity context. The steps are as follows:

[0068] First, take the mask vector Z′ of the word representation Z′ [MASK] , after the MLP linear layer, we get Z″ [MASK] , used to represent the category information of the prompt prediction, that is, the prompt semantics.

[0069] Secondly, the vector representation R of global semantics and the vector representation Z″ of hint semantics are[MASK] As input, the gated fusion module is used to extract key semantic information. Based on the gating mechanism, this module dynamically adjusts the weights of different semantic sources to achieve effective fusion between global semantics (represented by R) and hint semantics. The specific operations are as follows:

[0070] (1) Use the first gated memory unit to map the global semantics R and the mask vector Z″ [MASK] Perform information filtering to determine how much mask semantics to retain, as follows:

[0071] Z″ [MASK] =MLP(Z′ [MASK] );

[0072] c=σ(W c (R+Z″ [MASK] )+b c );

[0073] C keep =(1-c)·Z″ [MASK] ;

[0074] C next =c·Z″ [MASK] ;

[0075] Where W c 、b c are the weight matrix and bias of the first gate unit respectively, c is the update gate, C keep Represented as the retained mask semantics (first mask semantics), C next Refers to the mask semantics flowing into the next gated neuron (second mask semantics).

[0076] (2) Through the second gated memory unit according to C next Determine how much global semantics the mask semantics need to be integrated to retain the required global semantics, as follows:

[0077] p=σ(W m (C next +R)+b m );

[0078] P choose =p·R;

[0079] Where W m 、b m are the weight matrix and bias of the second gating unit respectively, p represents the weight ratio of global semantics, P choose Represents global semantics.

[0080] The entity type prediction P is represented by the final semantics of the sentence, including C keep and P choose Two parts, as follows:

[0081] P=C keep +P choose +Z″ [MASK] .

[0082] In order to fully integrate the advantages of prompt tuning and fine-tuning, this embodiment uses a gated memory module to fuse the semantics obtained by the two methods. The predicted value of each candidate instance is calculated and the cross entropy operation is performed with the true value to obtain the training loss as follows:

[0083]

[0084] Where Loss is the training loss function, n is the number of entities, and y i is the true value of the i-th entity, P i is the predicted value of the i-th entity.

[0085] Through the above method, fine-tuning and hint tuning are combined, and the goals of fine-tuning and pre-training tasks are optimized, which can stimulate the potential of pre-training in multiple dimensions, combine the semantics of different tuning, and obtain multiple semantic representations. It realizes the dynamic semantic association between the hint template and the instance global context and mask, makes full use of global semantics and local semantic information, and effectively reduces the interference of other candidate entities on the current recognition results.

[0086] To verify the effectiveness of the method in this embodiment, training was performed on the public English datasets ACE2005 and GENIA, and several existing named entity recognition models based on deep learning were selected for comparison. Precision (P), recall (R), and F1 value were used as evaluation indicators to verify the performance of this model, as shown in Table 1.

[0087] Table 1 Training results of different models

[0088]

[0089] Compared to other existing methods, the proposed method effectively improves the accuracy of named entity recognition, while also achieving good performance on both English and Chinese datasets. This demonstrates that the introduced gated fusion mechanism can dynamically integrate information from different semantic sources, significantly enriching the semantic representation of each candidate entity and reducing the impact on the current entity recognition. This makes it suitable for situations where similar candidate entities exist in the context, especially for the recognition of nested entities.

[0090] Therefore, the present invention adopts the above-mentioned named entity recognition method of the fusion tuning mode, which can fully integrate the semantics obtained by extraction fine-tuning and prompt tuning, enrich the semantic representation of each candidate entity, and overcome the problems existing in the prior art.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A named entity recognition method integrating tuning, characterized in that: include: S1. Use the boundary detection model to identify the starting and ending boundaries of the candidate entity, obtain the set of all candidate entities, and set two labels to insert on both sides of the candidate entity to obtain the sentence marked with the candidate entity; S2. The obtained candidate entities are combined into a prompt template, and the prompt template and the sentences marked with the candidate entities are used to fill the input sequence, and the word representation is obtained through the pre-trained language model; S3. Based on the gating mechanism, deep learning networks are used for training to extract semantics of different dimensions, including global semantics and mask semantics, and then predict named entities.

2. The method for named entity recognition using a fusion tuning method according to claim 1, characterized in that: S2, including the following steps: S21. After the input sequence passes through the word embedding layer and encoding layer of the pre-trained language model, the input sequence semantics is obtained, and the classifier is used to obtain the probabilities of different entity types, as follows: H = BERT(T); R=Classifier(H [CLS] ); Where T is the input sequence, H is the sequence semantics, R is the probability of different entity types, Classifier() is composed of a multi-layer perceptron, H [CLS] The output vector of the last layer of the sentence classification head CLS; S22, the self-attention mechanism, and BiLSTM form an interaction layer to convert the sequence semantics H into word representations as follows: α=Soft max((H·W q )(H·W k ) T ); Z=α·(H·W v ); Z′=BiLSTM(Z); Where Z' is the sequence vector representation, α represents the similarity between the query and the key, Z is the predicted sequence, and W q 、W k 、W v is the weight matrix of the self-attention mechanism.

3. The method for named entity recognition using a fusion tuning method according to claim 1, characterized in that: In S3, the deep learning network includes a first gated memory unit and a second gated memory unit; wherein the first gated memory unit filters information of different entity types to extract fine-tuned semantics; the second gated memory unit retains the global semantics required by the mask to extract the hint semantics.

4. The method for named entity recognition using a fusion tuning method according to claim 3, characterized in that: The first gated memory unit obtains the masked semantics of the sentence by filtering the global semantics and mask vector, as follows: WITH" [MASK] =MLP(Z′ [MASK] ); c=σ(W c (R+Z″ [MASK] )+b c ); C keep =(1-c)·Z″ [MASK] ; C next =c·Z″ [MASK] ; Where W c 、b c are the weight matrix and bias of the first gate control unit respectively, and c is the update gate; C keep Represented as the first mask semantics as part of the prediction result; C next is the second mask semantics, serving as the input of the second gated memory unit.

5. The method for named entity recognition using a fusion tuning method according to claim 3, characterized in that: The second gated memory unit determines the retained global semantics based on the second mask semantics, as follows: p=σ(W m (C next +R)+b m ); P choose =p·R; In the formula, p represents the weight ratio of global semantics, P choose represents mask semantics, R represents global semantics, and σ is the activation function.

6. The method for named entity recognition using a fusion tuning method according to claim 1, characterized in that: In S3, the predicted named entities are represented as: P=C keep +P choose +Z [ ” MASK] ; Where P is the predicted entity type.

7. The method for named entity recognition using a fusion tuning method according to claim 1, characterized in that: In S3, predicting named entities involves using a gated memory mechanism to calculate the predicted value of each candidate instance and performing a cross entropy operation with the true value to obtain the training loss, as follows: Where Loss is the training loss function, n is the number of entities, and y i is the true value of the i-th entity, P i is the predicted value of the i-th entity.

Images