Two-stage contrast learning training method of named entity recognition model and related device

Abstract

The invention belongs to the field of natural language processing, and relates to a two-stage contrast learning training method of a named entity recognition model and a related device. The method comprises the steps of obtaining a first training text statement; processing through a pre-trained first named entity recognition model to obtain a classification result of whether the information is an entity vector or not; marking the first training text statement with the entity vector to obtain a second training text statement; processing through a pre-trained second named entity recognition model to obtain an entity type of a second training text statement; according to the comparison distance between the prototype vector of each entity category label and the corresponding second training word feature and the comparison distance between the second training word features, two-stage comparison learning loss is obtained through calculation; adjusting parameters of the named entity recognition model based on the two-stage contrast learning loss, and then continuing to train to obtain a trained second named entity recognition model; according to the method, the classification difficulty of each step is reduced through a two-stage named entity recognition model. By means of two-stage contrast learning, the misjudgment rate during model classification is reduced.

Description

Technical Field

[0001] The invention belongs to the field of natural language processing, and in particular relates to a two-level contrastive learning training method and a related device for a named entity recognition model. Background Art

[0002] Named Entity Recognition (NER) is a core subtask in the field of Natural Language Processing (NLP), which aims to identify and classify predefined entities such as names, places, dates, etc. from text. This technology is crucial to improving the performance of downstream tasks such as machine translation and text summarization because it can extract key information from text. However, the NER task itself faces a major challenge, which requires a large amount of manually annotated data to train the model to obtain good classification performance. In practical applications, due to cost considerations and time and human resource constraints, many fields cannot obtain a large amount of annotated data. Using only a small amount of data to train conventional models usually leads to model overfitting, that is, the model performs well on the training data but performs poorly on unseen data. Therefore, effectively performing NER tasks under the condition of limited annotated data is an urgent problem to be solved.

[0003] The named entity recognition model based on the BIO model is an important technology in the field of natural language processing (NLP). Figure 1 As shown in the figure, the traditional BIO-based named entity recognition model works as follows: First, use common pre-trained models (BERT, Robert, etc.) to segment and vectorize the sentences input by the user. The obtained feature vector is input into the SoftMax classifier to obtain the confidence of each token corresponding to the predefined category. The category with the highest confidence for each token is decoded to obtain the predicted category of each token. The obtained category information is integrated, for example, B-PER and all the consecutive I-PERs following it are regarded as a PER entity, so as to obtain the entity information in the sentence. However, the decoding method of the traditional BIO model is complex, and the model needs to learn strong semantic correlation to ensure the final classification performance. In the scenario of small samples, a small amount of data is not enough to support model training, which makes it difficult for the small sample named entity recognition model to classify.

[0004] Prototype networks are widely used in small-sample named entity recognition (NER) tasks, such as Figure 2As shown in the figure, the named entity recognition model of the prototype network works as follows: Train on the source dataset to obtain a general semantic space. Fine-tune on the target dataset using a small number of samples to obtain the final semantic space and prototype vector. Input the sentence into a pre-trained model such as BERT to obtain the vector representation of each token. Put the vector representation of the token into the feature space to calculate the distance with the cluster of each category, and finally complete the classification of each token. Decode and integrate the classified results to obtain the entity representation in the final sentence; the prototype network completes the classification through the prototype vector and feature space, reducing the dependence on semantics; its core idea is to use rich source domain data to build a general semantic space. In this space, entities of the same category will be embedded to form specific clusters, and the centers of these clusters are regarded as "prototype" representations to represent the labels of specific categories. When processing data in the target domain, a small number of samples will be mapped to this general semantic space, and new prototype representations will be calculated based on this. After that, new entities will be classified into the category represented by the nearest prototype based on their distance from these prototype representations.

[0005] However, in the process of applying prototype networks for small-sample named entity recognition, there are two main problems: one is the limitation of label semantic representation. Current methods often rely on label names or descriptions to enhance the semantic representation of labels and combine them with prototype representations to form the final label representation. However, such label information may be too brief and unable to provide sufficient contextual support, thus limiting the quality of label semantic representation. The second is classification errors caused by the close distance between labels. Although the purpose of the prototype network design is to reduce the distance between entities of the same type, in some cases, this approach may lead to blurred boundaries between labels of different subcategories. For example, when dealing with subcategories under the large category of "people" such as "novelists", "thinkers", and "mathematicians", because they share the same upper-level category, semantic overlap is prone to occur, which makes them too close in the semantic space, increasing the difficulty of correct classification. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a two-level contrastive learning training method and related devices for a named entity recognition model to improve the following problems:

[0007] The present invention adopts a two-stage named entity recognition model, using the first named entity recognition model to first identify whether a single training word feature is an entity vector, and then using the second named entity recognition model to further identify and classify, which can reduce the difficulty of the named entity recognition task, avoid the problem of redundant calculation of non-entities, and thus improve the overall accuracy of the named entity recognition model. The present invention uses a natural language set to replace the entity recognition label, so that a more reliable and accurate feature space and prototype vector can be initialized. It greatly helps the learning and generalization ability of the model in a small sample scenario and reduces the difficulty of prototyping labels. The present invention uses a two-level contrast loss based on the prototype vector to more effectively capture and distinguish inter-class information and reduce interference during model classification. The model performs well in small sample tasks, further improves the generalization ability and robustness of the model, and solves the problem of blurred boundaries.

[0008] In a first aspect of the present invention, the present invention proposes a two-stage contrastive learning training method for a named entity recognition model, the method comprising:

[0009] Obtaining a first training text sentence;

[0010] Processing the first training text sentence through a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier;

[0011] By labeling the first training text sentence having the entity vector, a second training text sentence is obtained; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence;

[0012] The second training text sentence is processed by the pre-trained second named entity recognition model to obtain the entity type of the second training text sentence; the second named entity recognition model includes a second encoder and a softmax layer; the prototype vector of each second entity category label and the second training word feature of the second training text sentence are extracted by the second encoder; the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text sentence;

[0013] The two-level contrastive learning loss is calculated by the contrastive distance between the prototype vector of each entity category label and the corresponding second training word feature, and the contrastive distance between each second training word feature;

[0014] After adjusting the parameters of the named entity recognition model based on the two-level contrastive learning loss, the training is continued until the model training is completed, thereby obtaining a trained second named entity recognition model.

[0015] In a second aspect of the present invention, the present invention further proposes a two-stage contrastive learning training device for a named entity recognition model, the device comprising:

[0016] A first acquisition module, used to acquire a first training text sentence;

[0017] A first recognition module is used to process the first training text sentence through a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier;

[0018] A first processing module is used to obtain a second training text sentence by labeling the first training text sentence having an entity vector; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence;

[0019] A second recognition module processes the second training text sentence through a pre-trained named entity recognition model to obtain an entity type of the second training text sentence; the named entity recognition model includes a second encoder and a softmax layer; the second encoder extracts a prototype vector of each second entity category label and a second training word feature of the second training text sentence; the softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text sentence;

[0020] The second processing module is used to calculate the two-level contrastive learning loss through the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, and the contrast distance between each second training word feature; the parameters of the named entity recognition model are adjusted based on the two-level contrastive learning loss, and the training is continued until the model training is completed, so as to obtain a trained named entity recognition model.

[0021] Beneficial effects of the present invention:

[0022] The present invention reduces the difficulty of classification at each step through a two-stage named entity recognition model, and does not generate redundant calculations, which can bring better performance and effects in small sample scenarios. The two-level comparative learning of the present invention can effectively learn information in two dimensions: word token and word token, and word token and prototype vector prototype, and can more effectively capture information between classes, making the constructed feature space and prototype vector more discriminative, alleviating confusion during model classification. The present invention uses natural language sets and integrates word feature information in some of the same entity category labels during reasoning. In small sample scenarios, it can add word information that is richer in semantics than the label to the feature space, making the feature space more reliable, mapping more accurate, and generalizing more powerful. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the traditional named entity recognition model based on the BIO model;

[0024] Figure 2 It is a structural diagram of the traditional prototype network-based named entity recognition model;

[0025] Figure 3 is a schematic diagram of the structure of a named entity recognition model according to an embodiment of the present invention;

[0026] Figure 4 It is a flow chart of a two-stage contrastive learning training method of a named entity recognition model according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of the pre-training process of the first named entity recognition model in an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of a process for extracting a prototype vector of a second entity category label according to an embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of a process for extracting a prototype vector of a second entity category label according to a preferred embodiment of the present invention;

[0030] Figure 8 It is a structural schematic diagram of a two-stage contrastive learning training device for a named entity recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Named Entity Recognition (NER) is a very important subtask in natural language processing. It can automatically identify and mark predefined entities in unstructured sentences, thereby extracting important information and structured text. However, deep learning models often require a large amount of labeled data, which is often not available in downstream tasks. However, the current research on small-sample named entity recognition methods is still shallow, and there are certain problems in classification performance, robustness, and feature space representation. The decoding method of the traditional BIO model is complex, and the model needs to learn strong semantic relevance to ensure the final classification performance. In the scenario of small samples, a small amount of data is not enough to support model training, resulting in difficulty in classification of small-sample named entity recognition models. The small-sample named entity recognition model based on the prototype network completes classification through prototype vectors and feature space, reducing the dependence on semantics, but there are still some problems. The process of prototype construction is very dependent on label semantics, but the label information is generally very brief, which will lead to the final generated feature space. The robustness and generalization are not strong, thus affecting the classification results. The number of non-entities in a sentence is usually greater than the number of entities. Measuring the distance between all tokens and the prototype vector will result in a lot of redundant calculations, which is obviously unreasonable. If the prototype is constructed too roughly, the boundaries between similar prototypes will often be blurred, resulting in semantic overlap or being too close in the feature space, which will interfere with the final classification. Classification is performed only through a single vector calculation, which places too high a demand on the model. Small samples often cannot meet the training requirements of the model.

[0034] Based on existing technology issues, such as Figure 3As shown, an embodiment of the present invention proposes a small sample named entity recognition model based on a prototype network and two-level contrastive learning, wherein the small sample named entity recognition model is composed of a first named entity recognition model and a second named entity recognition model. The model divides the named entity recognition task into two subtasks, namely, first using the first named entity recognition model to recognize entities and then using the second named entity recognition model to classify them. For example, assuming that Bob and Jason live in America are input into the first named entity recognition model, the corresponding entity vector can be determined, Bob (I) and (O) Jason (I) live (O) in (O) America (I), I represents an entity vector, and O represents not an entity vector; in this way, the task difficulty is reduced, making the model easier to fit in a small sample scenario, and then these entity vectors are input into the second named entity recognition model to further extract the entity vectors of Bob, Jason, and America; at the same time, a more accurate and stable prototype feature space and mapping are constructed based on contrastive learning to enhance the classification performance of the model; the prototype vector is initialized using a natural language word set and a support sample to obtain a prototype vector with richer semantics and more stable. In an embodiment of a two-stage contrastive learning training method for a named entity recognition model provided in the present application, entities are first recognized using a first named entity recognition model and then classified using a second named entity recognition model, thereby reducing the difficulty of the task and making the model easier to fit in a small sample scenario; at the same time, a more accurate and stable prototype feature space and mapping are constructed based on contrastive learning to enhance the classification performance of the model; the prototype vector is initialized using a natural language word set and support samples to obtain a prototype vector with richer semantics and more stability.

[0035] The following is an example of a two-level contrastive learning training method for a named entity recognition model provided by the present application. Please refer to Figure 4 As shown, Figure 4 : is a flowchart of an embodiment of a two-stage contrastive learning training method for a named entity recognition model provided by the present application, and the training method embodiment includes:

[0036] 101. Obtain a first training text sentence;

[0037] In this embodiment, the first training text sentence is an arbitrary text sentence without an entity location label and an entity recognition label.

[0038] Specifically, the first training text sentence is obtained from the message queue in the database, and the first training text sentence uploaded by the target object through the operation interface of the terminal device can be received, or the first training text sentence can be obtained by other means, such as crawling big data, etc., and no specific restrictions are made here. For example, the first training text sentence obtained is "Li Bai is a poet in the Tang Dynasty", and through subsequent processes it can be determined that "Li Bai" is an entity location and has an entity location label, and "Li Bai" is an entity category and has an entity category label.

[0039] 102. Processing the first training text sentence by a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier;

[0040] Specifically, the main purpose of step 102 is to convert the input first training text sentence into a vector representation, and use the first encoder of the first named entity recognition model pre-trained on a large corpus to perform certain word segmentation, and then obtain the word embedding vector corresponding to each word; by identifying the word embedding vector corresponding to each word through the first classifier, it is determined whether the current word embedding vector is an entity. If yes, the marking result is I, and if not, the marking result is O. The obtained result will be used in the second named entity recognition model to determine the specific category.

[0041] The first named entity recognition model has been pre-trained on a large-scale text dataset to identify whether the word features in the text are entity vectors. It can be BERT-based Models: such as BERT-NER, RoBERTa-NER, etc. These models use the powerful representation capabilities of BERT (Bidirectional Encoder Representations from Transformers) and its variants to achieve remarkable results in NER tasks; it can also be Transformers-based Models: including a variety of pre-trained models in Hugging Face's Transformers library, such as ELECTRA, T5, etc., which are also suitable for NER tasks; it can even be Specialized NER Models: such as spaCy's pre-trained models, these models may be optimized for specific languages ​​or fields, and there is no specific restriction here.

[0042] For example, the BERT model pre-trained on a large corpus is used as the encoder (with parameter θ1). After a certain word segmentation, the word embedding vector h corresponding to each word is obtained. i :

[0043]

[0044] in, represents the first encoder, the subscript θ1 represents the parameters of the first encoder, and x i Indicates inputting the first training text sentence.

[0045] Exemplarily, the word embedding vector h obtained by the first training word feature i Send it to a SoftMax function for binary classification:

[0046] p(x i )=softmax(Wh i +b)

[0047] Among them, W and b are the weight matrix and bias of the first classifier. Here we only do a simple binary classification, that is, to determine whether the current word embedding vector is an entity.

[0048] In some embodiments, Figure 5 As shown, the pre-training process of the first named entity recognition model includes:

[0049] 201. Obtain a third training text sentence carrying an entity position label; the entity position label is used to indicate whether a training word of the third training text sentence is an entity vector;

[0050] In an embodiment of the present invention, the third training text sentence is similar to the first training text sentence, but the first training text sentence does not have a corresponding entity position label, while the third training text sentence has a corresponding entity position label for identifying whether the corresponding training word is an entity vector; for example, the third training text sentence obtained is "Li Bai is a poet of the Tang Dynasty", then "Li Bai", "Tang Dynasty" and "poet" are entity vectors and have corresponding entity position labels.

[0051] 202. Extracting third training word features of the third training text sentence by a first encoder to obtain a third training word feature set;

[0052] In an embodiment of the present invention, the third training text sentence can be segmented by the first encoder in the first named entity recognition model to extract the word embedding vector corresponding to each word as the third training word feature, and these third training word features are combined to obtain a third training word feature set.

[0053] 203. Identify each training word feature in the third training word feature set by a first classifier to obtain a classification result of whether the third training word feature is an entity vector;

[0054] In an embodiment of the present invention, the first classifier in the first named entity recognition model can be used to perform binary classification on each training word feature in the extracted third training word feature set. Exemplarily, the softmax function is used as the first classifier as an example, and each third training word feature in the third training word feature set is binary classified through the softmax function, and the third training word feature is linearly transformed. This is usually done through a fully connected layer (or linear layer), which maps the third training word feature to a new vector space, and the dimension of this new vector is equal to the number of categories of the classification task. For the binary classification problem, the dimension of the output vector is 2. The new vector after the linear transformation is sent to the SoftMax function. The SoftMax function converts each element of the new vector after the linear transformation into a probability value, so that the sum of all probabilities is 1. For the binary classification problem, SoftMax outputs two probability values, corresponding to the probabilities of the two categories, that is, the probability of whether the current third training word feature is an entity vector. If it is 1, it indicates that it is an entity vector, and if it is 0, it indicates that it is not an entity vector.

[0055] 204. Calculate a cross entropy loss based on a classification result of whether the third training word feature is an entity vector and an entity position label of the third training text sentence;

[0056] In an embodiment of the present invention, a loss function (such as cross entropy loss) is calculated based on the entity position label of the third training text sentence and the classification result of whether the third training word feature is an entity vector, and is used to update the model parameters.

[0057] 205. Pre-train the first named entity recognition model through the cross entropy loss.

[0058] In an embodiment of the present invention, the model parameters are updated based on the gradient of the loss function to minimize the loss. This can be achieved through a back propagation algorithm and an optimizer (such as SGD, Adam, etc.), so that the first named entity recognition model is pre-trained, so that the pre-trained first named entity recognition model can distinguish whether the word feature of the text sentence is an entity vector.

[0059] For example, the predicted result is compared with the actual result to calculate the loss L of the first stage CD :

[0060]

[0061] Among them, p(x i ) represents the prediction result of the i-th token, y i Indicates the true result, y i=0 means that the i-th token is not an entity or part of an entity, otherwise it means it is an entity. CD This loss is then used to optimize the parameters of the first named entity recognition model.

[0062] 103. Obtain a second training text sentence by labeling the first training text sentence having the entity vector; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence;

[0063] In the embodiment of the present invention, through step 101 and step 102, part of the training text sentences having entity vectors can be determined from the first training text sentences obtained in step 101, and this part of the training text sentences is used as the second training text sentences;

[0064] It should be noted that the processing process of the first named entity recognition model through steps 101 to 103 in the embodiment of the present invention can identify whether a single training word feature is an entity vector. This can reduce the difficulty of the named entity recognition task, avoid the redundant calculation problem of non-entities, and thus improve the overall accuracy of the named entity recognition model.

[0065] 104. Processing the second training text sentence by a pre-trained second named entity recognition model to obtain an entity type of the second training text sentence; the second named entity recognition model includes a second encoder and a softmax layer; extracting a prototype vector of each second entity category label and a second training word feature of the second training text sentence by the second encoder; calculating the distance between each second training word feature and the prototype vector of the corresponding second entity category label by the softmax layer to obtain the entity type of each second training word feature of the second training text sentence;

[0066] In the embodiment of the present invention, Figure 6 As shown, the process of obtaining the prototype vector of each second entity category label includes:

[0067] 401. Retrieve synonyms and / or sub-domain words of the second entity category label through a large language model or a dictionary;

[0068] In an embodiment of the present invention, for each predefined entity, a pre-trained large language model, such as BERT, GPT series (such as GPT-3, GPT-4), or a dictionary is selected to retrieve synonyms or words in a sub-field.

[0069] 402. Each of the second entity category labels and its synonyms and / or sub-domain words respectively constitutes a corresponding natural language word set;

[0070] In the embodiment of the present invention, a set S of natural language words can be constructed for them. i ={w1, w2, ..., w n}, used to calculate a more accurate and stable prototype vector. For example, the PER entity category can be represented by words such as human (synonym), John (subdomain word), etc.

[0071] 403. Using each word in the natural language word set as an expanded entity category label of the second entity category label;

[0072] In the embodiment of the present invention, the set S of natural language words i ={w1, w2, ..., w n Each word in} can represent an expanded specific category label.

[0073] 404. Pass the expanded entity category label of the second entity category label through the second encoder respectively to obtain a prototype vector of the second entity category label.

[0074] In an embodiment of the present invention, the second encoder in the second named entity recognition model can perform word segmentation processing on the expanded entity category label to extract the word embedding vector corresponding to each word, which is the prototype vector of the second entity category label.

[0075] Exemplarily, a natural language set is used to calculate a prototype vector for each label category, expressed as:

[0076]

[0077] The present invention uses a natural language word set to replace the original label, thereby initializing a more reliable and accurate feature space and prototype vector, which greatly helps the learning and generalization ability of the model in a small sample scenario.

[0078] In a preferred embodiment of the present invention, in order to obtain a more accurate prototype vector, the embodiment also uses the entity category label of the target domain to adjust the prototype vector of the second entity category label, thereby constructing a prototype vector that is more consistent with the distribution of the target domain.

[0079] Among them, Figure 7 As shown, the process of obtaining the prototype vector of each second entity category label also includes:

[0080] 411. Acquire a plurality of fourth training text sentences carrying fourth entity category labels; the fourth entity category labels are used to indicate the category of fourth training words in the fourth training text sentences;

[0081] 412. Pass the fourth training text sentence through a second encoder to obtain a fourth training word feature of the fourth training text sentence;

[0082] 413. If the second entity category label is the same as the fourth entity category label, superimpose the fourth training word feature and the prototype vector of the second entity category label to obtain an updated prototype vector of each second entity category label.

[0083] Exemplarily, the calculation method of the updated prototype vector of each second entity category label is expressed as:

[0084]

[0085] in, represents the prototype vector of the updated second entity category label of the jth category, λ1 is a hyperparameter, n represents the number of words in the natural language word set of the second entity category label, and w i The i-th word in the natural language word set representing the second entity category label; Represents the second encoder, Z j It refers to the words whose fourth entity category label of the jth category of the fourth training text sentence is the same as the second entity category label, and z represents Z j The word in .

[0086] In the embodiment of the present invention, the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text sentence. The calculation formula used is expressed as follows:

[0087]

[0088] Among them, y pred The entity type representing each second training word feature of the second training text sentence, h i represents the second word training word feature, represents the updated prototype vector of the second entity category label of the j-th category, and T represents all categories.

[0089] 105. Calculate a two-level contrastive learning loss by comparing the distance between the prototype vector of each entity category label and the corresponding second training word feature, and the distance between each second training word feature;

[0090] In the embodiment of the present invention, contrastive learning needs to construct positive and negative sample pairs. Here, in each batch during the training process, there will be multiple training data. The training data in the same batch are regarded as a set D. Then the data with the same label in the set are positive samples for each other, and the data with different labels are negative samples for each other. After obtaining the positive and negative sample pairs, two-level contrastive learning is designed based on token and token and token and prototype. Their formulas are expressed as follows:

[0091]

[0092]

[0093] Among them, L t represents the contrastive learning between tokens, that is, the contrastive distance between the features of each second training word, h + Indicates that in the current batch set D, i Positive samples with the same label, h j On the contrary. p represents the contrastive learning between token and prototype vector, that is, the contrastive distance between the prototype vector of each entity category label and the corresponding second training word feature, p i Indicates the prototype to which the current token belongs, p j represents all other prototypes except the one to which it belongs. τ is a scalar temperature parameter that controls the sharpness of the softmax distribution. Si m Represents a global similarity measure, which not only considers the similarity of individual words or phrases, but also their distribution and contextual relationship in the entire text. Common similarity measures such as cosine similarity, Jaccard similarity coefficient, and Euclidean distance can be used.

[0094] 106. After adjusting the parameters of the named entity recognition model based on the two-stage contrastive learning loss, continue training until the model training is completed, thereby obtaining a trained second named entity recognition model.

[0095] In some preferred embodiments, the distance loss can also be used to adjust the parameters of the named entity recognition model and continue training; the distance loss is calculated by the absolute distance between the prototype vector of each entity category label and the corresponding second training word feature; the parameters of the second named entity recognition model are adjusted based on the two-level contrastive learning loss and the distance loss, and then training is continued until the model training is completed to obtain a trained named entity recognition model.

[0096] For example, in addition to the contrastive learning loss, the absolute distance between the token and the prototype needs to be calculated and also included in the final loss, expressed as:

[0097]

[0098] d=(dis(h i , p1), dis(h i , p2), ..., dis(h i , p n ))

[0099] In the embodiment of the present invention, each loss function is used to quantify the degree of inconsistency between the model prediction result and the true value. When the prediction effect of the model is very good, the loss value will be very low; conversely, when the prediction effect is very poor, the loss value will be very high. Through the above-mentioned two-level contrast loss function and distance loss function, the second named entity recognition model can be optimized and trained to obtain a trained second named entity recognition model.

[0100] References Figure 8 A two-stage contrastive learning training device for a named entity recognition model is shown, the device comprising:

[0101] A first acquisition module, used to acquire a first training text sentence;

[0102] A first recognition module is used to process the first training text sentence through a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier;

[0103] A first processing module is used to obtain a second training text sentence by labeling the first training text sentence having an entity vector; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence;

[0104] A second recognition module processes the second training text sentence through a pre-trained named entity recognition model to obtain an entity type of the second training text sentence; the named entity recognition model includes a second encoder and a softmax layer; the second encoder extracts a prototype vector of each second entity category label and a second training word feature of the second training text sentence; the softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text sentence;

[0105] The second processing module is used to calculate the two-level contrastive learning loss through the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, and the contrast distance between each second training word feature; the parameters of the named entity recognition model are adjusted based on the two-level contrastive learning loss, and the training is continued until the model training is completed, so as to obtain a trained named entity recognition model.

[0106] ① The present invention proposes a two-level contrastive learning based on the prototype network, which can more effectively capture and distinguish inter-class information, initialize more discriminative prototype vectors, and enhance the classification performance, robustness and generalization of the model in small sample scenarios.

[0107] ② The entire two-stage model structure of the fusion prototype network proposed in the present invention reduces the difficulty of the model task and avoids redundant calculations.

[0108] ③ The method proposed in this invention to enrich the prototype vector by natural language word set and support set enriches the information source of the prototype vector in the scenario of small samples, which helps to calculate a more reliable and semantically stronger prototype vector.

[0109] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0110] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A two-stage contrastive learning training method for a named entity recognition model, characterized in that: The method comprises: Obtaining a first training text sentence; Processing the first training text sentence through a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier; By labeling the first training text sentence having the entity vector, a second training text sentence is obtained; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence; The second training text sentence is processed by the pre-trained second named entity recognition model to obtain the entity type of the second training text sentence; the second named entity recognition model includes a second encoder and a softmax layer; the prototype vector of each second entity category label and the second training word feature of the second training text sentence are extracted by the second encoder; the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text sentence; The two-level contrastive learning loss is calculated by the contrastive distance between the prototype vector of each entity category label and the corresponding second training word feature, and the contrastive distance between each second training word feature; After adjusting the parameters of the named entity recognition model based on the two-level contrastive learning loss, the training is continued until the model training is completed, thereby obtaining a trained second named entity recognition model.

2. The two-stage contrastive learning training method for a named entity recognition model according to claim 1, characterized in that: The pre-training process of the first named entity recognition model includes: Acquire a third training text sentence carrying an entity position label; the entity position label is used to indicate whether a training word of the third training text sentence is an entity vector; Extracting third training word features of the third training text sentence by the first encoder to obtain a third training word feature set; Identify each training word feature in the third training word feature set by a first classifier, and obtain a classification result of whether the third training word feature is an entity vector; Calculate the cross entropy loss by using the classification result of whether the third training word feature is an entity vector and the entity position label of the third training text sentence; The first named entity recognition model is pre-trained through the cross entropy loss.

3. The two-stage contrastive learning training method for a named entity recognition model according to claim 1, characterized in that: The process of obtaining the prototype vector of each second entity category label includes: Retrieving synonyms and / or sub-domain words of the second entity category label through a large language model or dictionary; Each of the second entity category labels and its synonyms and / or sub-domain words respectively constitutes a corresponding natural language word set; Using each word in the natural language word set as an expanded entity category label of the second entity category label; The expanded entity category labels of the second entity category labels are respectively passed through the second encoder to obtain prototype vectors of the second entity category labels.

4. The two-stage contrastive learning training method for a named entity recognition model according to claim 3, characterized in that: The process of obtaining the prototype vector of each second entity category label also includes: Acquire a plurality of fourth training text sentences carrying fourth entity category labels; the fourth entity category labels are used to indicate the category of fourth training words of the fourth training text sentences; Passing the fourth training text sentence through a second encoder to obtain a fourth training word feature of the fourth training text sentence; If the second entity category label is the same as the fourth entity category label, the fourth training word feature is superimposed on the prototype vector of the second entity category label to obtain an updated prototype vector of each second entity category label.

5. The two-stage contrastive learning training method for a named entity recognition model according to claim 4, characterized in that: The calculation method of the updated prototype vector of each second entity category label is expressed as: in, represents the prototype vector of the updated second entity category label of the jth category, λ1 is a hyperparameter, n represents the number of words in the natural language word set of the second entity category label, and w i The i-th word in the natural language word set representing the second entity category label; Represents the second encoder, Z j It refers to the words whose fourth entity category label of the jth category of the fourth training text sentence is the same as the second entity category label, and z represents Z j The word in .

6. A two-stage contrastive learning training method for a named entity recognition model according to claim 1 or 5, characterized in that: The calculation formula used by calculating the distance between each second training word feature and the prototype vector of the corresponding second entity category label through the softmax layer to obtain the entity type of each second training word feature of the second training text sentence is expressed as: Among them, y pred The entity type representing each second training word feature of the second training text sentence, h i represents the second word training word feature, represents the updated prototype vector of the second entity category label of the j-th category, and T represents all categories.

7. The two-stage contrastive learning training method for a named entity recognition model according to claim 1, characterized in that: The method further comprises: The distance loss is calculated by the absolute distance between the prototype vector of each entity category label and the corresponding second training word feature; After adjusting the parameters of the second named entity recognition model based on the two-stage contrastive learning loss and the distance loss, training is continued until the model training is completed to obtain a trained named entity recognition model.

8. A two-stage contrastive learning training device for a named entity recognition model, characterized in that: The device comprises: A first acquisition module, used to acquire a first training text sentence; A first recognition module is used to process the first training text sentence through a pre-trained first named entity recognition model to obtain a classification result of whether the first training word feature is an entity vector; the first named entity recognition model includes a first encoder and a first classifier; A first processing module is used to obtain a second training text sentence by labeling the first training text sentence having an entity vector; the second training text sentence has a second entity category label, and the second entity category label is used to indicate the category of the training word of the second training text sentence; A second recognition module processes the second training text sentence through a pre-trained named entity recognition model to obtain an entity type of the second training text sentence; the named entity recognition model includes a second encoder and a softmax layer; the second encoder extracts a prototype vector of each second entity category label and a second training word feature of the second training text sentence; the softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text sentence; The second processing module is used to calculate the two-level contrastive learning loss through the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, and the contrast distance between each second training word feature; the parameters of the named entity recognition model are adjusted based on the two-level contrastive learning loss, and the training is continued until the model training is completed, so as to obtain a trained named entity recognition model.

Images