A partial label named entity recognition method based on a pre-trained model

Abstract

The application discloses a partial label named entity recognition method based on a pre-training model, and comprises the following steps: inputting a predicted sentence into an overall model to obtain partial entity words and entity type labels; combining the partial entity words at the input end of the original predicted sentence to form a word sequence, and obtaining a head position sequence, a tail position sequence and a word label corresponding to the word sequence; performing sequence encoding on the obtained entity word token sequence and word label sequence through BERT in a partial label small model to obtain corresponding sequence encoding vectors, and then splicing the sequence encoding vectors to obtain a fused feature vector; performing feature fusion on the obtained feature vector through a transformer feature extractor in the partial label small model in combination with corresponding position encoding to obtain a final feature vector representation; and performing label prediction and decoding on the obtained fused feature through CRF. The method adopts a pre-training large model+small model mode to replace external dictionary information, and can effectively perform named entity recognition and extraction.

Description

Technical Field

[0001] This invention relates to the field of named entity recognition technology in the field of information extraction, specifically to a partial label named entity recognition method based on a pre-trained model. Background Technology

[0002] Named entity recognition (NER) is a fundamental task in natural language processing (NLP), aiming to determine the semantic categories of named entities in a given text, such as organizations, names of people, and locations. As a cornerstone of NLP tasks, NER plays a crucial role in many downstream tasks, such as relation extraction and question answering.

[0003] Currently, named entity recognition methods include dictionary-based or rule-based methods, machine learning-based methods, and deep learning semantic encoder-based methods.

[0004] Rule-based extraction methods identify and extract entities by matching text features with customized rule features. These methods offer good flexibility and are easy to operate. They utilize features such as punctuation marks, keywords, and indicator words, employ expert-constructed rule templates, and identify entities through pattern and string matching.

[0005] Dictionary-based methods were initially used in conjunction with rule-based extraction methods. Dictionary-based entity recognition refers to the process of attentional matching between each word in the dictionary and the document being processed. This method requires constructing a specific dictionary, matching the text to be recognized against the constructed dictionary according to the requirements, and continuously refining the dictionary in the process.

[0006] The traditional named entity recognition (NER) methods described above utilize prior information from an entity dictionary to match potential entity words in a sentence, and then use manually summarized rules to filter and determine the named entity type. The combination of dictionary and rules makes NER more flexible and accurate. The advantages of this method are its speed, high accuracy, and suitability for small-scale text recognition. The disadvantages of dictionary- and rule-based entity recognition methods are that their extraction efficiency and accuracy are far higher than manual extraction, but they require domain experts to define custom rules, resulting in excessive time and effort. Different entity types require customized rules, leading to poor portability.

[0007] Statistical machine learning methods start from a given, labeled training set, manually construct features, and label each word in the text according to a specific model to achieve named entity recognition. After using machine learning methods, domain experts no longer need to manually build templates or rules, but rely on annotated corpora to train models.

[0008] In machine learning-based named entity recognition methods, labeled words are typically represented using a label set; therefore, machine learning-based methods are also called sequence labeling methods. Representative statistical machine learning-based entity recognition techniques include Hidden Markov Models, Conditional Random Fields, Maximum Entropy Models, and Support Vector Machines.

[0009] With the improvement of computing power, deep learning has gradually emerged, and numerous named entity recognition methods based on deep learning have appeared, becoming the mainstream method for named entity recognition. Its advantage lies in its ability to automatically learn syntactic and dictionary features from text, thereby greatly improving recognition accuracy. These named entity recognition algorithms basically introduce deep neural networks (DNNs) as semantic encoders for feature extraction, replacing the original complex and cumbersome feature engineering.

[0010] In addition to encoding the semantic information of the input sentence, external auxiliary information is also incorporated, expanding the structure of deep learning-based named entity recognition models. For Chinese named entities, dictionary information is the most widely used type of external information, providing lexical-level information and lexical boundaries, helping to improve the accuracy of named entity recognition. To effectively integrate lexical information into the model, the FLAT model was proposed, which uses a clever positional encoding based on Transformer to fuse dictionary information. This significantly improves the accuracy of Chinese named entity recognition.

[0011] While deep learning neural network models that incorporate dictionary information can improve entity extraction performance, they often fail to function without dictionary information, and the use of dictionary information also limits the application scenarios of the models. Furthermore, due to the differences between the domain dictionaries of the source and target domains, the dictionary information learned by the model trained in the source domain cannot be transferred to the target domain, resulting in poor domain adaptability of the neural network model. Summary of the Invention

[0012] The purpose of this invention is to address the shortcomings of existing technologies by proposing a partial-label named entity recognition method based on a pre-trained model, employing a "large model + small model" design approach. This method uses a pre-trained large model + small model approach to replace external dictionary information, mitigating the problem of dictionary information discrepancies. The large pre-trained model obtains lexical-level hints for corresponding sentences to aid in named entity recognition.

[0013] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0014] A partially labeled named entity recognition method based on a pre-trained model includes the following steps:

[0015] S1. Input the predicted statement into the large pre-trained model UIE in the overall model that has been pre-trained, and obtain some entity words and entity type labels. The overall model includes a large pre-trained model and some small label models, wherein the large pre-trained model is the UIE model.

[0016] S2. Partial entity words are concatenated to the end of the input in the original prediction statement to form a word sequence. The corresponding head and tail position sequences and word labels are obtained from this word sequence. Finally, the corresponding head and tail position sequences and word label sequences are combined in a planar lattice structure to obtain complete lattice structure data. This lattice structure data is then input into a Transformer-based partial label mini-model for encoding and prediction. Each cell in the lattice structure data is defined as a span, which encompasses characters, entity tokens, entity labels, head tags, and tail tags.

[0017] S3. In the partial label mini-model, the obtained word token sequence and word label sequence are respectively encoded by BERT to obtain the corresponding sequence encoding vectors, and then concatenated to obtain the fused feature vector;

[0018] S4. In the partially labeled small model, the obtained feature vectors are fused by combining the corresponding positional encoding with the transformer feature extractor to obtain the final feature vector representation.

[0019] S5. Use the obtained fused features to perform label prediction and decoding via CRF.

[0020] Preferably, step S1 is implemented as follows:

[0021] The predicted sentence S is input into the UIE model, and the UIE model with large pre-training in this paper is used to initially identify some entity words E in the sentence. p ={e p，1 e p，2 , ...e p，m}, E p ∈E={e1, e2, ... e n}, m < n. Where E is the complete set of entities in the fully annotated corpus, and n is the number of entities in the Grand Truth dataset, i.e., E. Finally, we obtain partial entity words and their type labels as lexical-level prompts for the partial label mini-model, W = [{e...]. p，1 ,t1}...{e p，m , t m}], e p，m For the identified entity words, t m Tag it.

[0022] Preferably, step S2 is implemented as follows:

[0023] The W-word tag set obtained from the initial prediction in step S1 is concatenated into the original sentence S = (s1, s2, ... s) of length l. l The end of ) . Thus we obtain a new word sequence representation S′=(s1, s2, ... s l e p，1 e p，2 , ...e p，m The length of the predicted sentence is L = l + m, where l is the length of the original predicted sentence and m is the sequence length of the partial entity words. We then represent the word sequence S′ using lattice-structured data as G, and input it into the partial label mini-model.

[0024] Preferably, the word token sequence processing procedure in step S3 is as follows:

[0025] For the word sequence token = (g1...g1) in the lattice structure data representation G obtained in step S2... L In the partially labeled small model, we use BERT to encode word sequence vectors to obtain the word sequence encoding vector H = (h1...h2). L H∈R L×d L is the length of the word sequence, and d is the vector dimension.

[0026] Preferably, in step S3, the word tag sequence processing method is as follows:

[0027] First, for a word tag set E of length K... T = (t1, t2, ..., t3) K K is the number of entity label categories in the dataset; combining the B, I, and U labels, the number of categories of B-* and I-* is related to the entity label set E. T Since the number of entity tags is the same, there are a total of 3K+1 entity tag relationships, including K B-*, I-*, and K entity category tags* and 1 U tag, resulting in the word tag set E. T′ = (r1, r2, ..., r 3K+1 Then, manually complete the following: the tag "BD" can be represented as "beginof drug"; the tag "ID" can be represented as "inside of drug"; the tag "U" can be represented as "unknown", and "D" can be manually completed as "drug". The converted entity tags better reflect the natural semantics of the entity tags.

[0028] Secondly, the natural word tag set E T′The input is encoded in the BERT of the partial labeling mini-model. For each natural entity label's BERT output, the token sequence vector of [CLS] represents the semantic features of that entity label. Finally, the feature sequence of the word label is represented as E. T′ ∈R (3K+1)×d ;

[0029] For a corpus S′ of word sequence length L, the label is tag={t1,t2,...t L For the label t1(BD) of character s1, we obtain the corresponding label sequence vector by mapping the natural word label set ET′. Finally, the word label sequence is represented as tag∈R L×d L is the length of the word label sequence, and d is the vector dimension.

[0030] As a preferred embodiment, the method for obtaining the fused feature vector of word sequence vector and tag sequence vector in step S3 is as follows:

[0031] The word feature vector H obtained in step S3 is aligned and concatenated with the word tag feature vector tag to obtain the final feature vector representation H. e ,

[0032] Preferably, the method for obtaining the position code in step S4 is as follows:

[0033] In the lattice structure data, the head position and tail position represent the position index of the first and last character of the character token in the original corpus, respectively. For a token that represents only one character, its head and tail are the same.

[0034] This paper employs a planar lattice structure method to fit prior entity information and its entity label type information using relative position encoding after head-tail position transformations. For two lattices i and j, there are three forms—intersection, inclusion, and separation—depending on their positions in the corpus. Relationship modeling is achieved by utilizing the transformation of head and tail position information between lattices, resulting in the relative position encoding vector POS. ij ∈R L×d L is the word sequence length, and d is the vector dimension.

[0035]

[0036] Among them W r It is a learnable parameter, h i and t j Let P represent the beginning and end positions of the i-th and j-th characters, respectively, and let ⊕ denote the concatenation operation.

[0037]

[0038]

[0039] Where P span In this context, "span" is represented as POS. ij h i -h j ,h i -t j , t i -h j and t i -t j k is the position-encoded index;

[0040] d model =F×d head

[0041] F represents the number of heads to note, d head Note the size of each head.

[0042] Preferably, the method for obtaining the final feature vector representation in step S4 by combining relative position information is as follows:

[0043] The final fused word representation H obtained in step S3 e The input is fed into the transformer encoder in the partially labeled mini-model to perform feature fusion and extraction based on relative position information. Finally, we obtain the sequence vector H after feature fusion. end H end =transformer(H e POS).

[0044] Preferably, the specific method of step S5 is as follows:

[0045] The original character length of the feature vector obtained in step S4 is input into the CRF for label prediction to obtain the final character label. Finally, the predicted label is decoded, and the entity is constructed by searching from the character starting with B-* to the character before the first U label.

[0046] Preferably, in step S1, the pre-training method for the overall model is as follows:

[0047] S1-1. Fine-tuning the large pre-trained UIE model using fully labeled corpus: First, convert the fully labeled corpus into the format required by the large pre-trained model, and then use the converted data as input to fine-tune the large pre-trained model.

[0048] S1-2. Using a fully labeled corpus, and employing the Mask concept from BERT, the entity words and their type labels in the Mask part are used to obtain dataset D through data augmentation.mask ;

[0049] S1-3, Transfer dataset D mask The sentences are represented in a case structure manner, i.e., case structure sentences, and then input into the partial label model small model proposed in this paper for training, so that the partial label model can obtain the ability to supplement entity words;

[0050] S1-4. Encode the case structure sentence into a sequence vector using BERT in the partial label small model, and concatenate the word sequence vector and the label sequence vector as a fusion vector.

[0051] S1-5. The fused vector is processed by fusing location information through the transformer feature extractor in the partially labeled small model to obtain a feature vector containing location information.

[0052] S1-6. Input the original character length of the feature vector obtained in step (5) into the CRF for label prediction, calculate the loss and train iteratively.

[0053] Preferably, the loss calculation method in steps S1-6 is as follows:

[0054] In a partial labeling model, to calculate the score Sc(X, y) of the output tag sequence y corresponding to statement S,

[0055]

[0056] Where A is the tag transition probability matrix, and P i，yi Represents a single character w i Mapping to tags yi The non-normalized probability, where n is the original length of the character sequence, is used to define a probability value for each correct tag sequence y using the softmax function. x Represents all tag sequences,

[0057]

[0058] During training, we need to maximize the likelihood probability, using log-likelihood.

[0059]

[0060] Finally, the loss function in CRF is used to calculate the loss, which is defined as follows:

[0061] LOSS = -log(p(y|X)).

[0062] Preferably, the Bert model is set to "bert-base-chinese", with an epoch of 100, a learning rate of 0.05, and the learning rate is progressively adjusted using the SGD optimizer, while the model is trained using CRF loss.

[0063] This invention has the following characteristics and beneficial effects:

[0064] The above technical solution first fine-tunes the large pre-trained model using a small batch of fully labeled data. Then, it uses the fine-tuned pre-trained model to obtain word-level prompts for some entities. Finally, it uses a Transformer-based partial labeling mini-model to further optimize the entity extraction results. The partial labeling mini-model uses a flat-lattice structure to integrate the input text and the partial entity and word label information identified by the pre-trained language model. It uses BERT for encoding, extracts feature representations using Transformer, and inputs them into the CRF layer to predict entity labels. This alleviates the problem of dictionary information differences between different domains, obtains corresponding sentence prompts at the lexical level through the large pre-trained model, and enhances the accuracy of named entity recognition. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of the operation of the present invention.

[0067] Figure 2 This is a diagram of the planar lattice structure data structure of the present invention.

[0068] Figure 3 This is a natural language conversion diagram of the label in this invention.

[0069] Figure 4 This is a schematic diagram of the overall model structure of the present invention.

[0070] Figure 5 This is a flowchart of the pre-training process of the present invention.

[0071] Figure 6 This is a schematic diagram of the Mask corpus of the present invention. Detailed Implementation

[0072] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0073] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0074] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0075] This invention provides a partially labeled named entity recognition method based on a pre-trained model, such as... Figure 1 As shown, it includes the following steps:

[0076] Step (1) Input the predicted statement S into the pre-trained overall model to obtain the partial named entity vocabulary E. p The entity type labels are combined to form a vocabulary label set W. The overall model includes a large pre-trained model UIE and several smaller label models.

[0077] Step (2) concatenates the words in the vocabulary tag set W from step (1) at the end of the original sentence S input to form a word sequence S′, and obtains the head and tail position sequences and corresponding tag sequences corresponding to the word sequence. Finally, the complete lattice structure sentence representation input G is obtained by combining them in a planar lattice structure manner and inputting it into the partial label small model. Each lattice is defined as a set of spans, which cover characters or entity words tokens, entity tags, head tags and tail tags.

[0078] Step (3) involves performing sequence encoding on the word token sequence and the processed word tag sequence in the lattice structure data obtained in step (2) using BERT to obtain the corresponding sequence encoding vector H and tag, and then concatenating them to obtain the fused feature vector H. e .

[0079] Step (4) involves fusing the feature vectors from step (3) using the transformer feature extractor in the partially labeled small model in conjunction with relative position encoding (POS).

[0080] Step (5) uses CRF to predict and decode the feature vectors obtained in step (4).

[0081] Furthermore, the specific implementation process of step (1) is as follows:

[0082] Step (1) Input the predicted statement S into the large pre-trained UIE model, and use the large pre-trained model to initially identify some entity words E in the statement. p ={e p，1 e p，2 , ...e p，m}, E P ∈E={e1, e2, ... e n}, m < n. Where E is the complete set of entities in the fully annotated corpus, and n is the number of entities in the Grand Truth dataset, i.e., E. Finally, we obtain partial entity words and their type labels as lexical-level prompts for the partial label mini-model, W = [{e...]. p，1 ,t1}...{e p，m , t m}], e p，m For the identified entity words, t m Tag it.

[0083] Furthermore, the specific implementation process of step (2) is as follows:

[0084] Step (2) concatenates the W-word tag set obtained from the preliminary prediction in step (1) into the original sentence S = (s1, s2, ... s) with a character length of l. l The end of ) . Thus we obtain a new word sequence representation S′=(s1, s2, ... s l e p，1 e p，2 , ...e p，m The length of the predicted sentence is L = l + m, where l is the length of the original predicted sentence and m is the sequence length of the partial entity words. We then represent the word sequence S′ using lattice-structured data as G, such as... Figure 2 As shown.

[0085] For clarity, this embodiment provides a specific example. The input sentence S is "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconite, etc." A pre-trained model obtains a vocabulary tag set W of [{musk, I}, {aconite, I}], where I represents the entity tag components. The entity words from the vocabulary tag set are appended to the end of the original input sentence to obtain "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconite, etc., musk and aconite." Finally, the planar grid structure formed by combining the beginning and end position information and the tag information is as follows: Figure 2 As shown in the figure. The tags are represented by B, I, and U. B indicates that the character element is the beginning of an entity, I indicates that the character element is the middle part of an entity fragment, and U indicates that the character element is not entity content. B-* and I-* indicate that the character element belongs to the beginning or middle part of an entity of category *. This contains character-level fine-grained entity type information.

[0086] Furthermore, the specific implementation process of step (3) is as follows:

[0087] Step (3) involves further processing of the lattice structure data G obtained in step (2).

[0088] 3-1 For the word sequence token = (g1...g1) in G, L In the partially labeled small model, we use BERT to encode word sequence vectors to obtain the word sequence encoding vector H = (h1...h2) / (h1...h3) / (h1...h4) / (h1...h5) / (h1...h6) / (h1...h7) / (h1...h8) / (h1...h9 ... L H∈R L×d L is the length of the word sequence, and d is the vector dimension.

[0089] 3-2 For the label sequence tag = (t1...t2) in G, L To enable the model to perceive entity labels and better integrate label information with word features, the model needs to distinguish the semantics of entity labels. Therefore, we need to convert meaningless B, I, and U labels into natural semantic labels with contextual meaning.

[0090] 3-2-1 First, for a word tag set E of length K... T = (t1, t2, ..., t3) K K represents the number of entity label categories in the dataset. Combining the B, I, and U labels, the number of categories for B-* and I-* is related to the entity label set E. T Since the number of entity tags is the same, there are a total of 3K+1 entity tag relationships, including K B-*, I-*, and K entity category tags* and 1 U tag, resulting in the word tag set E. T′ = (r1, r2, ..., r 3K+1), then we manually complete it. The label "B-D" can be expressed as "begin of drug". The label "I-D" can be expressed as "inside of drug". The label "U" can be expressed as "unknown", and we manually complete "D" as "drug", so that the converted entity labels can better reflect the natural semantics of entity labels.

[0091] 3-2-2 Next, we take the natural word label set E T′ input into the BERT pre-training model in the partial label small model for encoding. For the BERT output of each natural entity label, the token sequence vector of <[BOS_never_used_51bce0c785ca2f68081bfa7d91973934]> is used to represent the semantic feature of this entity label. Finally, the feature sequence of word labels is represented as E T′ ∈R (3K+1)×d , and the overall process is shown in Figure 3 .

[0092] 3-3 For a corpus S' with a word sequence length L, the label is tag={t1, t2, ...t L}, for the label t1(B-D) of the character s1, we obtain the corresponding label sequence vector by mapping and querying the natural word label set E T′ Finally, the word label sequence is represented as tag∈R L×d , where L is the length of the word label sequence and d is the vector dimension. Then it is aligned and spliced with the word feature H to obtain the final word representation H e ,

[0093] Further, the specific implementation process of step (4) is as follows:

[0094] Input the final fused word representation H obtained in step (3) into the transformer encoder in the partial label small model to perform feature fusion and extraction combined with relative position information. The head position and tail position in the grid structure data respectively represent the position indices of the first and last character of the character token in the original corpus. For a token that only represents one character, its head position and tail position are the same. e

[0095] In this embodiment, a specific case is given: for the character element "云" in "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconitum, etc., thymus aconitum", its head and tail positions in the word sequence S' are the same, both 1. For the entity word "musk", its first character position in the original sequence S' is 11, and the last character position is 12.

[0096] ​To enable prior entities obtained in the pre-trained model to interact with the input sentence, we use a planar lattice structure method to fit the prior entity information and its entity label type information using relative position encoding after head and tail position transformations. For two lattices i and j, there are three forms—intersection, containment, and separation—depending on their positions in the corpus. Relationship modeling is achieved by utilizing the transformation of head and tail position information between lattices. The relative position encoding vector POS is used. ij ∈R L×d L is the length of the word sequence, and d is the vector dimension.

[0097]

[0098] Among them W r It is a learnable parameter, h i and t j This represents the beginning and end positions of the i-th and j-th characters, respectively. ⊕ represents the concatenation operation. Where P is:

[0099]

[0100]

[0101] Where P span In this context, "span" is represented as POS. ij h i -h j ,h i -t j , t i -h j and t i -t j k is the positional encoding index.

[0102] d model =F×d head

[0103] F represents the number of heads to note, d head Note the size of each head.

[0104] Finally, we obtain the sequence vector H after feature fusion. end H end =transformer(H e POS).

[0105] Furthermore, the specific implementation process of step (5) is as follows:

[0106] The feature vector H obtained in step (4) endThe original character length is input into the CRF for label prediction to obtain the final character label. For example, the original length of the feature vector for the sentence "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconite, etc." is "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconite, etc." Finally, the predicted tags are decoded by searching for characters starting with B-* up to the character preceding the first U tag to form the entity.

[0107] Reference Figure 5 This is a flowchart of the pre-training process of the present invention. The pre-training process of the large pre-trained UIE model and the partially labeled small model of the present invention includes the following steps:

[0108] Step (1) Fine-tuning the large pre-trained UIE model using a fully annotated corpus.

[0109] Step (2) Using a fully labeled corpus, the Mask concept of BERT is adopted, and the entity words and their type labels in the Mask part are obtained through data augmentation to obtain dataset D. mask .

[0110] Step (3) involves taking the dataset D from step (2). mask The sentences are represented using a case structure and then input into a partial labeling mini-model for training, enabling the partial labeling mini-model to acquire the ability to supplement entity words.

[0111] Step (4) encodes the lattice-structured sentence input in step (3) using BERT in the partial label mini-model. The concatenated word sequence vector and label sequence vector are used as the fusion vector.

[0112] Step (5) extracts and fuses the fused feature vector from step (4) by fusing location information through the transformer feature extractor in the partially labeled small model.

[0113] Step (6) Input the original character length of the feature vector obtained in step (5) into the CRF for label prediction, calculate the loss and perform iterative training.

[0114] Furthermore, the specific implementation process of step (1) is as follows:

[0115] For fully annotated corpora, we convert them to the corresponding data format using the data processing methods of the UIE model, and then use small batches of data to fine-tune the large pre-trained UIE model.

[0116] Furthermore, the specific implementation process of step (2) is as follows:

[0117] For the fully annotated corpus, we expand and obtain dataset D using the Mask concept from BERT, which masks partial entity words and their type labels, through data augmentation. mask .

[0118] For example, such as Figure 6 As shown in (a), for the ingredient list "Yunnan Baiyao: Panax notoginseng, musk, aconite, etc.", two entities, musk and aconite, can be randomly selected as partial labels. The entity label of Yunnan Baiyao is masked, i.e., the label is U, resulting in "Yunnan Baiyao ingredients: Panax notoginseng, musk, aconite, etc., musk and aconite" as expanded training data. Figure 6 In (b), if the entities “Yunnan Baiyao” and “Musk” are randomly selected as partial labels, then “Yunnan Baiyao ingredients: Panax notoginseng, musk, Aconitum carmichaelii and other Yunnan Baiyao musk” will be obtained as the expanded training data.

[0119] Furthermore, the specific implementation process of step (3) is as follows:

[0120] The dataset D obtained in step (2) mask The data is represented using a lattice structure and then input into a partially labeled mini-model for training. The specific lattice structure representation is described in detail in step (2) of the workflow of this invention.

[0121] Furthermore, the specific implementation process of step (4) is as follows:

[0122] The word sequences and tag sequences in the lattice structure data obtained in step (3) are encoded. The specific operation is described in detail in step (3) of the working process of this invention.

[0123] Furthermore, the specific implementation process of step (5) is as follows:

[0124] The fused feature vector obtained in step (4) and the calculated relative position encoding vector are used for feature fusion and extraction in the transformer encoder of the partial label small model. Detailed information is provided in step (4) of the workflow of this invention.

[0125] Furthermore, the specific implementation process of step (6) is as follows:

[0126] The original character length of the feature vector obtained in step (5) is input into the CRF for label prediction to obtain the final character label. y = CRF(H end ).

[0127] In a partial label mini-model, we need to calculate the score Sc(X,y) of the output tag sequence y corresponding to statement S.

[0128]

[0129] Where A is the tag transition probability matrix, and P i，yi Represents a single character w i Mapping to tags yi The non-normalized probability. n is the original length of the character sequence. Using the softmax function, a probability value is defined for each correct tag sequence y, Y. x Represents all tag sequences.

[0130]

[0131] During training, we need to maximize the likelihood probability, using log-likelihood.

[0132]

[0133] Finally, the loss function in CRF is used to calculate the loss, which is defined as follows:

[0134] LOSS = -log(p(y|X))

[0135] The BERT model was selected as "bert-base-chinese", with epochs set to 100 and a learning rate of 0.05. The learning rate was progressively adjusted using the SGD optimizer, and the model was trained using CRF loss.

[0136] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for partial-labeled named entity recognition based on a pre-trained model, characterized in that, Includes the following steps: S1. Input the predicted statement into the large pre-trained model UIE in the pre-trained overall model to obtain some entity words and entity type labels. The overall model includes the large pre-trained model and some small label models. S2. Concatenate some entity words to the end of the input in the original prediction statement to form a word sequence, and obtain the head and tail position sequences and word labels corresponding to the word sequence. Finally, combine the corresponding head and tail position sequences and corresponding word label sequences in a planar lattice structure to obtain complete lattice structure data. Then, input the lattice structure data into a Transformer-based partial label small model for encoding and prediction. Each cell in the lattice structure data is defined as a span, which covers characters, entity words, tokens, entity labels, head tags, and tail tags. S3. In the partial label small model, the obtained entity word token sequence and word label sequence are respectively sequence encoded by BERT to obtain the corresponding sequence encoding vector, and then concatenated to obtain the fused feature vector; S4. In the partially labeled small model, the obtained feature vectors are fused by combining the corresponding positional encoding with the transformer feature extractor to obtain the final feature vector representation. S5. Use the obtained fused features to perform label prediction and decoding via CRF.

2. The method for partial label named entity recognition based on a pre-trained model according to claim 1, characterized in that, In step S2, the length of the word sequence is L = l + m, where l is the length of the original prediction statement and m is the sequence length of the partial entity words. The word tags are represented by B, I, and U. B indicates that the character element is the beginning of an entity, I indicates that the character element is the middle part of an entity fragment, and U indicates that the character element is not entity content. B-* and I-* indicate that the character element belongs to the beginning and middle part of an entity of category *, which contains character-level fine-grained entity type information.

3. The method for partial label named entity recognition based on a pre-trained model according to claim 2, characterized in that, In step S3, the word tag sequence processing method is as follows: First, for a word tag set E of length K... T = (t1, t2, ..., t3) K K is the number of entity label categories in the dataset; combining the B, I, and U labels, the number of categories of B-* and I-* is related to the entity label set E. T Since the number of entity tags is the same, there are a total of 3K+1 entity tag relationships, including K B-*, I-*, and K entity category tags* and 1 U tag, resulting in the word tag set E. T ′=(r1, r2, ..., r 3K+1 Then, manually complete the following: the tag "BD" can be represented as "beginof drug"; the tag "ID" can be represented as "inside of drug"; the tag "U" can be represented as "unknown", and "D" can be manually completed as "drug". The converted entity tags better reflect the natural semantics of the entity tags. Secondly, the natural word tag set E T The input is fed into BERT for encoding. For each natural entity label's BERT output, the token sequence vector of [CLS] represents the semantic features of that entity label. Finally, the feature sequence of the word label is represented as E. T ′∈R (3K+1)×d ; For a corpus S′ of word sequence length L, the label is tag={t1,t2,...t L For the label t1(BD) of character s1, we query the natural word label set E through mapping. T We obtain the corresponding tag sequence vector. Finally, the word tag sequence is represented as tag∈R L×d L is the length of the word label sequence, and d is the vector dimension.

4. The method for partial label named entity recognition based on a pre-trained model according to claim 3, characterized in that, The method for obtaining the position code in step S4 is as follows: In the lattice structure data, the head position and tail position represent the position index of the first and last character of the character token in the original corpus, respectively. For a token that represents only one character, its head and tail are the same. This paper employs a planar lattice structure method to fit prior entity information and its entity label type information using relative position encoding after head-tail position transformations. For two lattices i and j, there are three forms—intersection, inclusion, and separation—depending on their positions in the corpus. Relationship modeling is achieved by utilizing the transformation of head and tail position information between lattices, resulting in the relative position encoding vector POS. ij ∈R L ×d L is the word sequence length, and d is the vector dimension. Among them W r It is a learnable parameter, h i and t j This indicates the beginning and end positions of the i-th and j-th characters, respectively. This represents the concatenation operation, where P is: Where P span In this context, "span" is represented as POS. ij h i -h j h i -t j , t i -h j and t i -t j k is the position-encoded index; d model =F×d head F represents the number of heads to note, d head Note the size of each head.

5. The method for partial label named entity recognition based on a pre-trained model according to claim 4, characterized in that, The specific method for step S5 is as follows: The original character length of the feature vector obtained in step S4 is input into the CRF for label prediction to obtain the final character label. Finally, the predicted label is decoded, and the entity is constructed by searching from the character starting with B-* to the character before the first U label.

6. The method for partial label named entity recognition based on a pre-trained model according to claim 1, characterized in that, In step S1, the pre-training method for the overall model is as follows: S1-1. Fine-tune the large pre-trained model using a fully annotated corpus; S1-2. Using a fully labeled corpus, and employing the Mask concept from BERT, the entity words and their type labels in the Mask part are used to obtain dataset D through data augmentation. mask ; S1-3, Transfer dataset D mask The sentences are represented in a case structure manner, i.e., case structure sentences, and then input into a partial label small model for training, so that the partial label small model can acquire the ability to supplement entity words; S1-4. Encode the case structure sentence into a sequence vector using BERT, and concatenate the word sequence vector and the label sequence vector as a fusion vector; S1-5. The fused vector is processed by the transformer feature extractor to extract and fuse the location information to obtain a feature vector containing the location information. S1-6. Input the original character length of the feature vector obtained in step (1-5) into the CRF for label prediction, calculate the loss and perform iterative training.

7. The method for partial label named entity recognition based on a pre-trained model according to claim 6, characterized in that, In step S1-1, the fine-tuning method is as follows: first, the fully annotated corpus is converted into the format data required by the large pre-training model, and then the converted data is used as the input of the large pre-training model to fine-tune the large pre-training model.

8. The method for partial label named entity recognition based on a pre-trained model according to claim 7, characterized in that, In steps S1-6, the loss calculation method is as follows: In the small model, to calculate the score Sc(X, y) of the output tag sequence y corresponding to statement S, Where A is the tag transition probability matrix, and P i，yi Represents a single character w i Mapping to tags yi The non-normalized probability, where n is the original length of the character sequence, is used to define a probability value for each correct tag sequence y using the softmax function. x Represents all tag sequences, During training, we need to maximize the likelihood probability, using log-likelihood. Finally, the loss function in CRF is used to calculate the loss, which is defined as follows: LOSS = -log(p(y|X)).

9. A partially labeled named entity recognition method based on a pre-trained model according to claim 7, characterized in that, The overall model is selected as "bert-base-chinese", with an epoch of 100 and a learning rate of 0.

05. The learning rate is progressively adjusted using the SGD optimizer, and the model is trained using CRF loss.

Images