New word named entity recognition system based on template contrast learning

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a new word named entity recognition system based on template contrast learning. The system comprises a template set generation module and a context comparison learning module. The template set generation module is used for generating a template set with high quality and rich semantics by using a large language model, and merging sentence-level information from various templates by using a pooling method designed for a plurality of templates; and the context comparison learning module is used for finely adjusting context representation and improving the recognition performance of the neologism entity by utilizing context information of the entity. According to the method, sentence level features are introduced to expand previous text fragment representation, so that classification features contain certain explicit-structured context information; the introduced additional sentence level context information features are finely adjusted by applying a template-based contrast learning strategy, and are introduced into the training of a main task as an auxiliary task for multi-task learning, so that more performance gains are obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and knowledge engineering, and particularly relates to a new word named entity recognition system based on template contrast learning. Background Art

[0002] Named entity recognition (NER for short in English) is a quite fundamental task in natural language processing, and it plays an important role in many downstream tasks such as knowledge graph construction, information retrieval, question answering systems, and recommendation systems. With the continuous development of Internet technology, many applications are built on the basis of natural language processing, and these top-level designs often require the support of underlying technologies. Therefore, it is very necessary to deeply explore named entity recognition, especially named entity recognition in complex application scenarios. The NER task aims to identify the fragments of entities mentioned in an input sentence and further determine the type of the fragment entity. For example, given the input text “Aurora Couture aims to illuminate the fashion industry with its unique designs and commitment to ethical practices.”, the model needs to output the text fragment “Aurora Couture” as the entity fragment and assign its entity type as a brand entity.

[0003] In recent years, many deep learning models applied to named entity recognition have shown good results. However, the performance of these model methods is severely affected by the new word problem, that is, when many entity words have not appeared in the previous training data. This exposes the challenges brought by the new word problem to traditional named entity recognition and also restricts the further improvement of model performance. It may be because the fragment representation of the new word entity itself has not been fine-tuned during model training, and most of the previous model methods rely on the fine-tuned fragment representation to complete the named entity recognition task, so the recognition effect of the new word named entity is poor.

[0004] Previous model studies for the new word named entity recognition task mainly dealt with it through three methods: external knowledge, new word embedding, and context embedding. The method using external knowledge enhances the recognition ability of the model by introducing external information. However, in many cases, using external knowledge may not be convenient or cost-effective, especially for domain-specific expertise. The method using new word embedding attempts to construct robust embedding vectors for new words to handle the new word problem. However, such methods only focus on using the semantic information of the new words themselves without considering other useful information such as context-related or sentence-level information. Relatively speaking, the method using context embedding integrates the context information except the new word embedding itself to complete named entity recognition. Such methods generally attempt to learn the appropriate embeddings of the words near the new word entity for this purpose. However, these model methods do not fully exploit the sentence-level context information around the new word entity, and this feature can actually help to further improve the performance of new word named entity recognition. Summary of the Invention

[0005] The purpose of the present invention is to provide a new word named entity recognition system based on template contrast learning to efficiently complete the new word named entity recognition task.

[0006] The new word named entity task proposed by the present invention can be described as follows: Given a text sequence T, extract the entity fragment set E from it. Based on this pattern, the new word named entity recognition method based on template contrast learning proposed by the present invention can improve the performance of the new word named entity recognition task. Named entity recognition research is affected by the new word problem, that is, entities not encountered in the training scenario are encountered in the test scenario, resulting in unsatisfactory performance results. Affected by the information explosion in the information age, new word entities will continuously emerge in actual application scenarios. Therefore, the new word problem is relatively common. There are relatively few existing studies specifically for named entity recognition in the new word scenario. The new word scenario poses new challenges to existing named entity recognition research and also restricts the development of multiple important downstream applications. How to make good use of the special background of the new word scenario is the key to solving this problem. Existing studies have not paid good attention to this point, and the achieved results are not satisfactory enough.

[0007] The new word named entity recognition system based on template contrast learning provided by the present invention, abbreviated as ContextNER, has an architecture diagram as Figure 1As shown in the figure. The general framework for named entity recognition based on text fragments generally constructs specific text fragments to represent the recognition results obtained by the final classifier of the model. ContextNER introduces an additional sentence-level feature to expand the previous text fragment representation, so that the classification features contain certain explicitly constructed context information. At the same time, a template-based contrastive learning strategy is applied to fine-tune the introduced additional sentence-level context information features. The introduced contrastive learning strategy will make the context representation of the text fragment corresponding to the new word entity closer to the semantics of the template sentence formed by this entity and the correct entity type in the semantic space, and farther away from the semantics of the template sentence formed by this entity and the wrong entity type in the semantic space. And this contrastive learning strategy is introduced as an auxiliary task into the training of the main task for multi-task learning, so as to obtain more performance gains.

[0008] Specifically, the new word named entity recognition system based on template contrastive learning provided by the present invention includes two modules: a template set generation module and a context contrastive learning module, where:

[0009] Template set generation module: Use a large language model (LLM) [1] to generate a high-quality and semantically rich template set, and use a pooling method designed for multiple templates to merge the context information from various templates.

[0010] Context contrastive learning module: Introduce an effective contrastive learning strategy independent of the model architecture to fine-tune the context representation, so as to further improve the performance for new word named entity recognition.

[0011] The specific details of each part of the model are elaborated in detail below.

[0012] (1) Template set generation module;

[0013] According to the design of ContextNER, the prediction of the output label is based on the fragment representation with context representation. This indicates that better context representation may be beneficial to further improve the performance of the model for the named entity recognition task. The present invention proposes a contrastive learning strategy in the following, which is based on template learning to further tune the context representation to obtain a context representation that is more beneficial for the model to perform named entity recognition.

[0014] Specifically, the empty template is defined as T = "[SPAN]is a [TYPE] entity.", where the position of [SPAN] needs to be filled with the currently concerned fragment, and the position of [TYPE] needs to be filled with the fragment s iCorrect or incorrect entity type labels. Therefore, a filled template can be obtained by filling the correct label in [TYPE]. This template is correct in terms of intuitive semantics and thus serves as a positive sample in contrastive learning. A filled template can also be obtained by filling an incorrect label in [TYPE]. This template is incorrect in terms of intuitive semantics and thus serves as a negative sample in contrastive learning. To enable the encoder to more effectively understand the semantic information of the template, a natural language transformation mapping needs to be performed on the label itself before filling it into the empty template T. For example, the label LOC is transformed into the natural language "location". Then, the filled template is also encoded in the same BERT-like encoder to obtain a similar sentence representation. The positive sample template is encoded to obtain a positive template representation, and the negative sample template is encoded to obtain a negative template representation. The positive template directly represents what the correct entity type corresponding to the text segment si is. For example, "Shanghai is a location entity." directly represents that the text segment "Shanghai" is a location entity. Therefore, the context representation of the text segment s i should be relatively aligned and close to the context representation of the positive sample template in the semantic embedding space because they both contain similar semantics. The context representation of the text segment s i should be far from the context representation of the negative sample template in the semantic embedding space, indicating that they have no related semantic relationship. This perspective of understanding can help the model more accurately extract context representations that are beneficial for downstream named entity recognition. In addition, the negative sample template also implies an incorrect semantic understanding. Therefore, pulling the original context representation c away from the representation of the negative sample is a way for the model to correct its understanding. In the example sentences, it can be "Shanghai is not an entity." or "Shanghai is a person entity.", and the semantics of both sentences are obviously incorrect and should be contrary to the pre-training corpus of the pre-trained language model.

[0015] The present invention uses GPT-4 to generate 100 templates, and then fills them as described above, and selects 10 representative templates from them. Formally, t = {t 1 , t 2 , … t k} is the template set. Let and be the positive template set and the negative template set corresponding to the template t i respectively;

[0016] (II) Context contrastive learning module

[0017] Selecting the fragment-based named entity recognition as the base model of ContextNER is mainly for the following two reasons: (1) The fragment-based named entity recognition method can explicitly extract the information about the entity word itself, and this information is relatively easy to be used by the subsequent named entity recognition model. (2) Compared with some previous sequence labeling model methods for named entity recognition, it shows better generalization ability in the new word scenario. Specifically, it includes: word representation layer, fragment representation layer, context representation layer, fragment classification layer, and positive and negative sample contrast learning layer.

[0018] Starting from the input sequence, after passing through the word representation layer, each sequence element will be converted into the corresponding vector representation. After converting the input sequence into the corresponding vector through the word representation layer, the corresponding fragment representation is constructed, and then the context information is added to the fragment representation and the context representation is fine-tuned using the contrast learning strategy;

[0019] (2.1) Word representation layer

[0020] Starting from the input sequence, after passing through this layer, each sequence element will be converted into the corresponding vector representation for the convenience of subsequent matrix and vector calculations. In formal terms, let X = {x 1 , x 2 , …, x n} represent the current input sentence sequence, where x i , i = 1, 2, …, n each represents a sequence element, generally a character or a word, and n represents the sequence length. The representation of the sequence element x i can be obtained through the following formula:

[0021]

[0022] where f(·) is a non-contextual word embedding operation operator or a word embedding operation operator that can utilize context information, such as Glove [2] , ELMo [3] , BERT [4] . d is the dimension of the embedded element.

[0023] (2.2) Fragment representation layer

[0024] After converting the input sequence into the corresponding vector through the word representation layer, the corresponding fragment representation can be constructed for subsequent judgments. Given X, let S = {s 1 , s 2 , …, s m} be the corresponding fragment set, where s i = (b i , e i ), b i represents the serial number at the beginning of the fragment, and e iIndicates the serial number at the end of the segment, i = 1, 2, …, m, where m represents the total number of segments. For each s i , a label y i can be assigned to the text segment it represents to indicate whether the current segment is an entity and what type of entity it is if it is an entity. For example, assuming the current input sequence sentence is "Shanghai is quite good.", then it can be converted to the corresponding segment set:

[0025] S = {(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (3, 3), (3, 4), (3, 5), (3, 6), (4, 4), (4, 5), (4, 6), (5, 5), (5, 6), (6, 6)};

[0026] In fact, there is only one location entity "Shanghai" in the original sentence, and its corresponding segment serial number is represented as (1, 2). Therefore, only the label corresponding to (1, 2) is LOC, and the labels of the remaining segments are all O, that is, they are not entities. For a segment s i = (b i , e i ), its constructed representation consists of two parts: boundary embedding and segment length embedding. The boundary embedding part consists of two parts: the representation of the segment start position element and the representation of the segment end position element . After concatenating the two representations, it is the boundary embedding part of the current segment representation

[0027]

[0028] The segment length embedding introduces the segment length feature, denoted as . The learning of the embedding feature in this part actually corresponds to the mapping from a certain set of positive integers to the embedding vector, which is randomly initialized during actual training, and then the corresponding feature representation can be learned through training. After concatenating the boundary embedding and the segment length embedding, the segment representation z i constructed by s i is obtained:

[0029]

[0030] (2.3) Context Representation Layer

[0031] The present invention uses BERT as an encoder to encode the entire input sentence, thereby obtaining a sentence-level representation. Given an input sequence sentence, the sentence representation c can be obtained using the following method:

[0032]

[0033] where h(·) represents the encoding operation function of the sentence encoder. The representation z i of the text segment s i consists of three parts: boundary embedding , segment length embedding and sentence representation c, as follows:

[0034]

[0035] Then, input z i with the same operation into the segment classification layer to predict the label s i of the corresponding text segment.

[0036] (2.4) Segment Classification Layer

[0037] The vector z i will be fed into a neural network of a multi-layer perceptron [5] . The purpose of this operation is to classify the segment s i into the entity types predicted by the model. Formally, denote this classification operation as the function F, and the function F maps the segment representation of dimension 3d + d′ to a vector of dimension , where represents the entire entity type space, and one more is because there is an additional non-entity type label O. The training loss of the model is:

[0038]

[0039] CrossEntropy(·,·) represents the cross-entropy loss function;

[0040] (2.5) Positive and Negative Sample Pairwise Contrastive Learning Layer

[0041] The templates used in the present invention are positive sample templates and negative sample templates respectively, is the set of positive sample templates for the text segment s i , is the set of negative sample templates for the text segment s i . After pooling each entity type set along the template dimension, the template representation of the entire template set is obtained. Use the following contrastive learning loss strategy:

[0042]

[0043] After combining the loss of the contrastive learning strategy discussed above, the loss function of the entire ContextNER is:

[0044]

[0045] Among them, λ is the weight of the contrastive learning loss. The larger the value of λ, the heavier the proportion of contrastive learning in the training process, and the greater the impact.

[0046] The present invention is based on the framework of the BERT series of models in the era of pre-trained language models. It models the named entity recognition problem as a classification problem based on text segments, and uses the features extracted by the pre-trained language model for downstream tasks. The present invention uses sentence-level context information as an auxiliary feature to enhance the model's recognition ability, and at the same time uses a template-based contrastive learning method to enable the model to learn better context information. By introducing the strategy of template contrastive learning, the new word named entity recognition task is optimized, especially with higher accuracy when dealing with unknown entities, and through enhancing context representation and multi-task learning, a significant improvement in performance is achieved. Brief Description of the Drawings

[0047] Figure 1 It is a schematic diagram of the new word named entity recognition framework based on template contrastive learning of the present invention.

[0048] Figure 2 It is a structural diagram of the encoder model used in the present invention. Detailed Embodiments

[0049] The superiority of the present invention will be further demonstrated below through specific implementation examples and comparative experiments.

[0050] (1) Evaluation Settings

[0051] The present invention evaluates the effect of new word named entity recognition, and still adopts the classic evaluation index at the entity level, F 1 . This evaluation index regards the result of the named entity recognition task as a classification task, and the category set is the classification result of the corresponding entity type. Specifically, for each sentence, the model may give the entity recognition results of some text segments or none of them are entities. At the same time, the entity labels will also tell the evaluation system which segments correspond to what type of entities. Under this evaluation criterion, the entity-level F 1 requires correct classification if and only if the predicted segment boundaries are accurate and the entity type classification is accurate. The number of predicted samples where the model predicts as an entity and the label is also an entity is denoted as TP, the number of predicted samples where the model predicts as an entity but the label is not the corresponding entity is FP, and the number of samples where the model does not predict as an entity but the label is actually an entity is FN. F 1 is the harmonic mean of precision and recall [6], incorporating both the evaluation metric of precision for assessing the accuracy of model judgments and the evaluation metric of recall for assessing the completeness of model judgments, can more comprehensively and specifically reflect the effectiveness of the model for this named entity recognition task.

[0052] (2) Hyperparameter Setting

[0053] The present invention selects BERT-large [4] as the base model of ContextNER, that is, using BERT-large as the text segment encoder f(·) and sentence encoder h(·) of ContextNER. Its structure diagram is as Figure 2 shown. For each input sentence sequence, the present invention limits its maximum length to 128 sequence elements. If the sentence sequence exceeds the length of 128 sequence elements, only the first 128 elements will be retained for calculation. In fact, enumerating all text segments has O(n2) number of segments for a sequence with length n, and the computational pressure is very large when n is relatively large. Therefore, the present invention limits the maximum length of text segments to k, because for the vast majority of named entities, the length of the entity itself is not very large, thus reducing the complexity of the number of segments to be calculated to O(nk). In the experiment, k = 4 is set, that is, the maximum length of the enumerated segments is 4. For the segment classification layer, a two-layer fully connected layer is used to implement. All experiments and models are carried out on an NVIDIA GeForce RTX 3090 graphics card. In addition, the large language model GPT-4 utilized by the present invention is the latest generation of natural language processing model developed by OpenAI, which represents an important milestone in the expansion of deep learning research. As part of the GPT series, GPT-4 inherits and extends the functions of its predecessors and has more powerful text generation, logical reasoning, and image recognition capabilities.

[0054] (3) Comparison Baseline

[0055] The present invention is compared with baselines, including BERT-Tagger, BERT-CRF, SpanNER, DataAug, InferNER, CoFEE, and MINER. BERT-Tagger predicts the entity categories of the tags in the text by adding a classification layer, thereby achieving efficient and accurate entity recognition. With the powerful context understanding ability of BERT, this method can achieve excellent recognition effects even with a small amount of labeled data, and has good cross-domain adaptability. BERT-CRF combines the pre-trained BERT model and conditional random field (CRF). First, BERT is used to obtain the context-related representation of each tag, and then the dependencies between tags are captured through the CRF layer, thereby achieving high-precision entity boundary and category prediction in the named entity recognition task. SpanNER focuses on locating entities by identifying spans in text, rather than relying on traditional tag sequence annotation, which can handle nested and overlapping entity structures more flexibly. This method usually scores text fragments of different lengths in combination with context information to determine which fragments constitute named entities. DataAug is a method that uses data augmentation technology to expand the training set of named entity recognition. It improves the generalization ability of the model and the recognition accuracy of rare entities by generating synthetic samples or variants. InferNER uses knowledge graphs and reasoning mechanisms to enhance the accuracy of entity recognition, and is particularly good at processing entities that are rare or have not appeared in the training data. CoFEE is a few-shot named entity recognition framework that uses pre-trained language models and meta-learning strategies to effectively extract new types of entities even with a small amount of annotated data. MINER automatically discovers and recognizes new entity types by mining patterns and contextual information in large-scale text data without the need for large amounts of manually annotated data.

[0056] (IV) Comparative experimental results and analysis

[0057] Table 1 shows the F performance of ContextNER and all baselines on five datasets. 1, where the highest score in each dataset is marked in bold. Compared with the baseline, ContextNER shows a significant performance improvement, especially on the WNUT2017 dataset with a 100% new word ratio. This indicates that ContextNER has better robustness in more complex new word scenarios. As the base model of the fragment-based NER model, SpanNER outperforms the two-sequence tagging models BERT-Tagger and BERT-CRF. DataAug enhances SpanNER through data augmentation, further improving the performance. MINER integrates the information bottleneck theory into SpanNER to better handle the new word named entity task, achieving an overall sub-optimal result. By fully utilizing sentence-level information, ContextNER surpasses all baselines, highlighting the key role of context information in new word named entity recognition.

[0058] Table 1

[0059]

[0060]

[0061] To verify the role of ContextNER, the present invention conducts an ablation study, and the results are shown in Table 2. The present invention compares the performance of three methods on the WNUT2017, JNLPBA, and TwitterNER datasets. The first method is the original SpanNER, the second is SpanNER + context representation, and the third is SpanNER + templatepooling & contrastive learning, that is, ContextNER. The ablation studies conducted on these three datasets consistently show that the components proposed by the present invention all have a positive impact on the performance of the model.

[0062] Table 2

[0063]

[0064]

[0065] ContextNER utilizes template-based contrastive learning and template pooling methods. To evaluate the performance of a single template on ContextNER, the present invention designs different templates. The results shown in Table 3 are obtained in the WNUT2017 dataset. From the experimental results, it is obvious that ContextNER is sensitive to template selection, and different templates directly affect the final performance. The template merging method based on multiple templates obtains the best result at the same time, which emphasizes the efficacy of template pooling.

[0066] Table 3

[0067]

[0068] The combination of context representation and its improvement through contrastive learning proposed by the present invention is model - independent. They can be inserted into the model using a sentence encoder. To verify this feature, the present invention compared variants of SpanNER and ContextNER with different pre - trained language models as encoders, including BERT - large, RoBERTa - large, and DeBERTa - large. The comparison results shown in Table 4 indicate that different pre - trained language models have different natural language understanding capabilities, leading to different NER performances. In addition, regardless of which pre - trained language model is used as the encoder, ContextNER is always better than SpanNER, proving the effectiveness of the components proposed by the present invention.

[0069] Table 4

[0070]

[0071]

[0072] References:

[0073] [1]OpenAI. 2023. GPT - 4 technical report. CoRR, abs / 2303.08774.

[0074] [2] Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named Entity Re- / Recognition as Span Prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7183–7195, Online. Association for Computational Linguistics. PENNINGTON J, SOCHER R, MANNING C. GloVe: Global vectors for word representation[C] / / Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP). Doha, Qatar: Association for Computational Linguistics, 2014:1532-1543.

[0075] [3] PETERS M E, NEUMANN M, IYYER M, et al. Deep contextualized word representations[C] / / Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long Papers). New Orleans, Louisiana: Association for Computational Linguistics, 2018:2227-2237.

[0076] [4]DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C] / / Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, 2019: 4171-4186.

[0077] [5]Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain[J]. Psychological review, 1958, 65(6): 386.

[0078] [6]Lewis D D. Evaluating text categorization i[C] / / Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991. 1991。

Claims

1. A new word named entity recognition system based on template contrastive learning, referred to as ContextNER, is characterized by: The new word named entity task is formulated as follows: given a text sequence T, a set of entity segments E is extracted from it; based on this model, the goal is to improve the performance of the new word named entity recognition task; first, an additional sentence-level feature is introduced to expand the previous text segment representation, so that the classification feature contains certain explicitly constructed context information; at the same time, a template-based contrastive learning strategy is applied to fine-tune the introduced additional sentence-level context information feature; The contrastive learning strategy introduced makes the contextual representation of the text fragment corresponding to the new word entity closer to the semantics of the template sentence composed of the entity and the correct entity type in the semantic space, and farther away from the semantics of the template sentence composed of the entity and the wrong entity type in the semantic space; and introduces the contrastive learning strategy as an auxiliary task into the training of the main task for multi-task learning, so as to obtain more performance gains; it includes two modules: template set generation module and context contrastive learning module, in which: Template collection generation module: It uses a large language model (LLM) to generate a high-quality and semantically rich template collection, and uses a pooling method designed for multiple templates to merge contextual information from various templates; Contextual Contrastive Learning Module: Introduces an effective contrastive learning strategy that is independent of the model architecture to fine-tune the context representation, thereby further improving the performance of named entity recognition for new words.

2. The new word named entity recognition system based on template contrast learning according to claim 1 is characterized in that: In the template set generation module, an empty template is defined as T = "[SPAN] is a [TYPE] entity.", wherein the position of [SPAN] is used to fill the currently focused segment, and the position of [TYPE] is used to fill the segment that is related to the currently focused segment. i Correct or incorrect entity type label; thus, by filling in the correct label in [TYPE], a filled template is obtained, which is intuitively semantically correct and thus serves as a positive sample in contrastive learning; by filling in the incorrect label in [TYPE], a filled template is obtained, which is intuitively semantically wrong and thus serves as a negative sample in contrastive learning; in order to enable the encoder to understand the semantic information of the template more effectively, a natural language conversion mapping is performed on the label itself before filling in the empty template T; then, the filled template is also encoded in the same encoder to obtain a similar sentence representation; the positive sample template is encoded to obtain the positive template representation, and the negative sample template is encoded to obtain the negative template representation; the positive template directly indicates what the correct entity type corresponding to the text fragment si is; thus, the text fragment s i The context representation of is aligned and close to the context representation of the positive sample template in the semantic embedding space; while the text fragment s i The contextual representation of the negative sample template is far away from the contextual representation of the negative sample template in the semantic embedding space, indicating that the two have no relevant semantic relationship; thus helping the model to more accurately extract the contextual representation that is beneficial to downstream named entity recognition.

3. The new word named entity recognition system based on template contrast learning according to claim 2 is characterized in that: The context contrastive learning module specifically includes: a word representation layer, a fragment representation layer, a context representation layer, a fragment classification layer, and a positive and negative sample contrastive learning layer; starting from the input sequence, each sequence element is converted to a corresponding vector representation through the word representation layer, and the input sequence is converted to a corresponding vector through the word representation layer to construct a corresponding fragment representation, and then context information is added to the fragment representation and the context representation is fine-tuned using a contrastive learning strategy; wherein: (1) Word Representation Layer Starting from the input sequence, after passing through the word representation layer, each sequence element is converted to the corresponding vector representation to facilitate the subsequent matrix and vector calculations; in formal definition, let X = {x1, x2, …, x n } represents the current input sentence sequence, where x i , i = 1, 2, ..., n each represents a sequence element, specifically a word or a term, and n represents the length of the sequence; for a sequence element x i The representation of is obtained by the following formula: Where f(·) is a non-contextual word embedding operator or a word embedding operator using contextual information; d is the dimension of the element after embedding, and g(·) is a neural network with context encoding capability; (2) fragment representation layer; After the input sequence is converted to the corresponding vector through the word representation layer, the corresponding fragment representation is constructed for subsequent judgment; given X, let S = {s1, s2, …, s m } is the corresponding fragment set, where s i =(b i ,e i ),b i Indicates the sequence number of the beginning of the fragment, e i Indicates the sequence number of the end of the fragment, i = 1, 2, ..., m, m represents the total number of fragments; for each s i , assign a label y to the text segment it represents i To indicate whether the current fragment is an entity, if so, what type of entity it is, then convert it to the corresponding fragment set: For a fragment i =(b i ,e i ), the representation it constructs consists of two parts: boundary embedding and fragment length embedding; the boundary embedding part consists of two parts: the representation of the element at the beginning of the fragment and the end position element of the fragment The concatenation of the two represents the boundary embedding part of the current fragment representation. The fragment length embedding introduces the fragment length feature, denoted as The embedding feature learning of this part actually corresponds to the mapping from a certain set of positive integers to the embedding vector. In actual training, it is randomly initialized, and then the corresponding feature representation is learned through the training number. After the boundary embedding and the fragment length embedding are spliced ​​together, s is obtained. i The constructed fragment represents z i : (2.3) Context Representation Layer Use BERT as an encoder to encode the entire input sentence to obtain a sentence-level representation; given an input sequence sentence, use the following method to get the sentence representation c: where h(·) represents the encoding operation function of the sentence encoder; the text segment s i The expression z i It consists of three parts: Boundary embedding , fragment length embedding And the sentence representation c is as follows: Then, z i The same operation is followed to input into the segment classification layer to predict the label s of the corresponding text segment. i ; (2.4) Fragment Classification Layer The vector z i Send it to a linear classification layer F(·) to classify the segment s i Classify it into the entity type predicted by the model; that is, record this classification operation as function F, which maps the fragment representation with dimension 3d+d′ to a fragment with dimension On the vector of represents the entire entity type space, +1 is due to the addition of a non-entity type label O; the training loss of the model is: Among them, CrossEntropy(·,·) represents the cross entropy loss function, (2.5) Positive and negative sample model comparison learning layer The templates are divided into positive sample templates and negative sample templates. For text fragments i The positive sample template set, For text fragments i A set of negative sample templates; after pooling each entity type set along the template dimension, the template representation of the entire template set is obtained; the following contrastive learning loss strategy is used: Combined with the loss of the contrastive learning strategy discussed above, the entire loss function is: Among them, λ is the weight of contrastive learning loss. The larger the λ value, the greater the proportion of contrastive learning in the training process, and the greater the impact.

Images