Two-stage contrastive learning training method for named entity recognition model and related device

Abstract

The application belongs to the field of natural language processing, and relates to a two-stage contrastive learning training method of a named entity recognition model and related devices.The method comprises the following steps: obtaining a first training text sentence;obtaining a classification result of whether it is an entity vector through processing by a pre-trained first named entity recognition model;labeling the first training text sentence with an entity vector to obtain a second training text sentence; obtaining an entity type of the second training text sentence through processing by a pre-trained second named entity recognition model; calculating a two-stage contrastive learning loss through a contrastive distance between a prototype vector of each entity category label and a corresponding second training word feature and a contrastive distance between each second training word feature; and continuing to train the named entity recognition model based on the two-stage contrastive learning loss to obtain a trained second named entity recognition model.The two-stage named entity recognition model reduces the classification difficulty of each step.The two-stage contrastive learning of the application reduces the misjudgment rate of the model during classification.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing, specifically relating to a two-level contrastive learning training method and related apparatus for a named entity recognition model. Background Technology

[0002] Named Entity Recognition (NER) is a core subtask in Natural Language Processing (NLP), aiming to identify and classify predefined entities, such as names of people, places, and dates, from text. This technology is crucial for improving the performance of downstream tasks such as machine translation and text summarization because it can extract key information from text. However, NER faces a major challenge: it requires a large amount of manually labeled data to train the model and achieve good classification performance. In practical applications, due to cost considerations and limitations in time and human resources, many fields cannot obtain large amounts of labeled data. Training a familiar model with only a small amount of data often leads to overfitting, meaning the model performs well on the training data but poorly on unseen data. Therefore, effectively performing NER tasks under limited labeled data conditions is a pressing problem to be solved.

[0003] Named entity recognition models based on the BIO model are an important technology in the field of natural language processing (NLP), such as... Figure 1 As shown, the traditional BIO-based named entity recognition model works as follows: First, a common pre-trained model (BERT, Robert, etc.) is used to segment and vectorize the user-input sentence. The resulting feature vectors are then input into a SoftMax classifier to obtain the confidence score of each token for a predefined category. The category with the highest confidence score for each token is then decoded to obtain the predicted category for each token. The obtained category information is then integrated; for example, B-PER and all subsequent I-PER are treated as a single PER entity to obtain the entity information in the sentence. However, the decoding method of the traditional BIO model is complex, requiring the model to learn strong semantic relevance to ensure the final classification performance. In scenarios with few samples, the limited data is insufficient to support model training, leading to difficulties in classification for named entity recognition models with few samples.

[0004] Prototype networks are widely used in few-shot named entity recognition (NER) tasks, such as... Figure 2As shown, the prototype network's named entity recognition model works as follows: A general semantic space is obtained by training on the source dataset. The final semantic space and prototype vectors are then fine-tuned using a small number of samples on the target dataset. Sentences are input into pre-trained models such as BERT to obtain vector representations of each token. The token vector representations are placed in the feature space, and distances are calculated between them and the clusters of each category, ultimately classifying each token. The classification results are decoded and integrated to obtain the final entity representation in the sentence. The prototype network performs classification using prototype vectors and the feature space, reducing its dependence on semantics. Its core idea is to utilize rich source domain data to construct a general semantic space. Within this space, entities of the same category are embedded to form specific clusters, and the centers of these clusters are considered "prototype" representations, used to represent labels of specific categories. When processing data from the target domain, a small number of samples are mapped to this general semantic space, and new prototype representations are calculated accordingly. New entities are then classified into the category represented by the nearest prototype based on their distance from these prototype representations.

[0005] However, applying prototype networks for few-shot named entity recognition faces two main problems: First, the limitations of label semantic representation. Current methods often rely on label names or descriptions to enhance the semantic representation of labels and combine them with prototype representations to form the final label representation. However, such label information may be too simplistic and fail to provide sufficient contextual support, thus limiting the quality of the label semantic representation. Second, classification errors caused by labels being too close together. Although prototype networks are designed to reduce the distance between entities of the same type, in some cases, this approach may lead to blurred boundaries between different subclass labels. For example, when dealing with subclasses under the broad category of "person," such as "novelist," "thinker," and "mathematician," semantic overlap is likely to occur because they share the same upper-level category. This makes them too close in semantic space, increasing the difficulty of correct classification. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a two-level contrastive learning training method and related apparatus for named entity recognition models to improve upon the following issues:

[0007] This invention employs a two-stage named entity recognition (NENT) model. The first NENT model identifies whether the features of a single training word constitute an entity vector, while the second NENT model further classifies and identifies the entity. This reduces the difficulty of the NENT task, avoids redundant computations on non-entity terms, and thus improves the overall accuracy of the NENT model. This invention uses a natural language set instead of entity recognition labels, enabling the initialization of a more reliable and accurate feature space and prototype vectors. This greatly enhances the model's learning and generalization capabilities in small-sample scenarios and reduces the difficulty of prototype label construction. This invention utilizes a two-stage contrastive loss based on prototype vectors to more effectively capture and distinguish inter-class information, reducing interference during model classification. This results in superior model performance in small-sample tasks, further improving the model's generalization ability and robustness, and resolving the issue of ambiguous boundaries.

[0008] In a first aspect, the present invention proposes a two-level contrastive learning training method for a named entity recognition model, the method comprising:

[0009] Obtain the first training text statement;

[0010] The first training text sentence is processed by the pre-trained first named entity recognition model to obtain the classification result of whether the features of the first training word are entity vectors; the first named entity recognition model includes a first encoder and a first classifier;

[0011] A second training text statement is obtained by labeling the first training text statement with entity vectors; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement;

[0012] The second training text statement is processed by a pre-trained second named entity recognition model to obtain the entity type of the second training text statement; the second named entity recognition model includes a second encoder and a softmax layer; the prototype vector of each second entity category label and the second training word features of the second training text statement are extracted by the second encoder; the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text statement;

[0013] The bi-level contrastive learning loss is calculated by the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature.

[0014] After adjusting the parameters of the named entity recognition model based on the two-level contrastive learning loss, the model continues to be trained until the training is completed, resulting in a second named entity recognition model that has been trained.

[0015] In a second aspect, the present invention also proposes a two-level contrastive learning training device for a named entity recognition model, the device comprising:

[0016] The first acquisition module is used to acquire the first training text statement;

[0017] The first recognition module is used to process the first training text statement through a pre-trained first named entity recognition model to obtain a classification result of whether the features of the first training words are entity vectors; the first named entity recognition model includes a first encoder and a first classifier;

[0018] The first processing module is used to obtain a second training text statement by labeling the first training text statement with entity vectors; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement;

[0019] The second recognition module processes the second training text statement using a pre-trained named entity recognition model to obtain the entity type of the second training text statement. The named entity recognition model includes a second encoder and a softmax layer. The second encoder extracts the prototype vector of each second entity category label and the second training word features of the second training text statement. The softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text statement.

[0020] The second processing module is used to calculate the bi-level contrastive learning loss by using the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature; and to continue training the named entity recognition model after adjusting the parameters based on the bi-level contrastive learning loss until the model training is completed, thus obtaining the trained named entity recognition model.

[0021] The beneficial effects of this invention are:

[0022] This invention reduces the classification difficulty at each step through a two-stage named entity recognition model, avoiding redundant computation and delivering better performance and results in scenarios with small sample sizes. The invention's two-level contrastive learning effectively learns information from two dimensions: word tokens and word tokens with prototype vectors. This more effectively captures inter-class information, making the constructed feature space and prototype vectors more discriminative and reducing confusion during model classification. Furthermore, this invention uses natural language sets and fuses word feature information from labels of the same entity category during inference. In scenarios with small samples, this adds richer word information to the feature space than the label semantics, making the feature space more reliable, the mapping more accurate, and the generalization ability stronger. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a traditional BIO-based named entity recognition model;

[0024] Figure 2 This is a schematic diagram of the structure of a traditional named entity recognition model based on prototype networks;

[0025] Figure 3 This is a schematic diagram of the structure of the named entity recognition model according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of the two-level contrastive learning training method of the named entity recognition model according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the pre-training process of the first named entity recognition model according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram illustrating the process of extracting the prototype vector of the second entity category label in an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram illustrating the process of extracting the prototype vector of the second entity category label according to a preferred embodiment of the present invention.

[0030] Figure 8 This is a schematic diagram of the two-level contrastive learning training device for the named entity recognition model according to an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Named Entity Recognition (NENT) is a crucial subtask in Natural Language Processing (NLP). It automatically identifies and labels predefined entities in unstructured sentences, enabling the extraction of important information and the creation of structured text. However, deep learning models often require large amounts of labeled data, which is frequently unavailable in downstream tasks. Current research on few-shot NENT methods is still in its early stages, exhibiting challenges in classification performance, robustness, and feature space representation. Traditional BIO models rely heavily on complex decoding methods, requiring the model to learn strong semantic relevance to ensure final classification performance. In few-shot scenarios, the limited data is insufficient to support model training, leading to difficulties in NENT classification. Prototype-based NENT models utilize prototype vectors and feature spaces for classification, reducing reliance on semantics, but still present challenges. Prototype construction heavily depends on label semantics, but label information is typically very brief, resulting in weak robustness and generalization of the generated feature space, thus affecting classification results. The number of non-entity elements in a sentence is usually greater than the number of entity elements. Measuring the distance between all tokens and their prototype vectors leads to a large amount of redundant computation, which is clearly unreasonable. Overly coarse prototype construction often results in blurred boundaries between similar prototypes, leading to semantic overlap or excessive proximity in the feature space, interfering with the final classification. Classifying based on a single vector computation places excessive demands on the model. Small sample sizes often fail to meet the model's training requirements.

[0034] Based on the problems of existing technologies, such as Figure 3As shown, this embodiment of the invention proposes a few-shot named entity recognition model based on prototype networks and two-level contrastive learning. The few-shot named entity recognition model consists of a first named entity recognition model and a second named entity recognition model. This model divides the named entity recognition task into two sub-tasks: first, the first named entity recognition model is used to identify entities, and then the second named entity recognition model is used to classify them. For example, assuming that "Bob and Jason live in America" ​​is input into the first named entity recognition model, the corresponding entity vectors can be determined: Bob(I)and(O)Jason(I)live(O)in(O)America(I), where I represents an entity vector and O represents a non-entity vector. This reduces the difficulty of the task and makes the model easier to fit in few-shot scenarios. These entity vectors are then input into the second named entity recognition model to further extract the entity vectors of "Bob", "Jason", and "America". At the same time, contrastive learning is used to construct a more accurate and stable prototype feature space and mapping, enhancing the classification performance of the model. The prototype vectors are initialized using a natural language vocabulary and support samples, thereby obtaining semantically richer and more stable prototype vectors. In an embodiment of the two-level contrastive learning training method for a named entity recognition model provided in this application, the first named entity recognition model is used to identify entities, and then the second named entity recognition model is used for classification. This reduces the difficulty of the task and makes the model easier to fit in small sample scenarios. At the same time, it also constructs a more accurate and stable prototype feature space and mapping based on contrastive learning, thereby enhancing the classification performance of the model. The prototype vector is initialized using a natural language vocabulary and support samples, thereby obtaining a prototype vector with richer semantics and greater stability.

[0035] The following will describe an embodiment of the two-level contrastive learning training method for a named entity recognition model provided in this application. Please refer to [link / reference]. Figure 4 As shown, Figure 4 This is a flowchart of an embodiment of a two-level contrastive learning training method for a named entity recognition model provided in this application. The training method embodiment includes:

[0036] 101. Obtain the first training text statement;

[0037] In this embodiment, the first training text statement is any text statement that does not have entity location labels and entity recognition labels.

[0038] Specifically, the first training text statement can be obtained from the message queue in the database, or it can be received from the first training text statement uploaded by the target object through the operation interface of the terminal device, or it can be obtained through other methods, such as crawling big data. No specific restrictions are made here. For example, if the obtained first training text statement is "Li Bai is a poet of the Tang Dynasty", subsequent processes can determine that "Li Bai" is an entity location with an entity location label and "Li Bai" is an entity category with an entity category label.

[0039] 102. The first training text sentence is processed by the pre-trained first named entity recognition model to obtain the classification result of whether the features of the first training words are entity vectors; the first named entity recognition model includes a first encoder and a first classifier;

[0040] Specifically, the main purpose of step 102 is to convert the input first training text statement into a vector representation. After segmentation using the first encoder of the first named entity recognition model pre-trained on a large corpus, the word embedding vector corresponding to each word is obtained. The word embedding vector is then identified by the first classifier to determine whether the current word embedding vector represents an entity. If it does, the result is labeled as 'I'; otherwise, it is labeled as 'O'. The obtained results are then used by the second named entity recognition model to determine the specific category.

[0041] The first named entity recognition model has been pre-trained on a large-scale text dataset to identify whether word features in the text are entity vectors. It can be BERT-based models, such as BERT-NER and RoBERTa-NER, which leverage the powerful representation capabilities of BERT (Bidirectional Encoder Representations from Transformers) and its variants, achieving significant results on NER tasks. It can also be Transformers-based models, including various pre-trained models from Hugging Face's Transformers library, such as ELECTRA and T5, which are also suitable for NER tasks. Furthermore, it can even be Specialized NER models, such as pre-trained models from spaCy, which may be optimized for specific languages ​​or domains; no specific limitations are imposed here.

[0042] For example, using BERT, a model pre-trained on a large corpus, as the encoder (with parameter θ1), after some word segmentation, the word embedding vector h corresponding to each word is obtained. i :

[0043]

[0044] in, This represents the first encoder, with the subscript θ1 representing the parameters of the first encoder, x i This indicates that the first training text statement is input.

[0045] For example, the word embedding vector h obtained from the features of the first trained word. i The data is fed into a SoftMax function for binary classification:

[0046] p(x i ) = softmax(Wh i +b)

[0047] Here, W and b are the weight matrix and bias of the first classifier. We'll only perform a simple binary classification here, determining whether the current word embedding vector represents an entity.

[0048] In some embodiments, such as Figure 5 As shown, the pre-training process of the first named entity recognition model includes:

[0049] 201. Obtain a third training text statement carrying entity location labels; the entity location labels are used to indicate whether the training words of the third training text statement are entity vectors;

[0050] In this embodiment of the invention, the third training text statement is similar to the first training text statement, but the first training text statement does not have a corresponding entity location label, while the third training text statement has a corresponding entity location label to identify whether the corresponding training word is an entity vector; for example, if the obtained third training text statement is "Li Bai is a poet of the Tang Dynasty", then "Li Bai", "Tang Dynasty", and "poet" are entity vectors and have corresponding entity location labels.

[0051] 202. Extract the third training word features of the third training text statement through the first encoder to obtain the third training word feature set;

[0052] In this embodiment of the invention, the first encoder in the first named entity recognition model can perform word segmentation on the third training text statement to extract the word embedding vector corresponding to each word, which is the third training word feature. These third training word features are combined to obtain the third training word feature set.

[0053] 203. Identify the features of each training word in the third training word feature set using the first classifier, and obtain the classification result of whether the third training word features are entity vectors;

[0054] In this embodiment of the invention, the first classifier in the first named entity recognition model can perform binary classification on each training word feature in the extracted third training word feature set. For example, the softmax function is used as the first classifier. The third training word features in the third training word feature set are binary classified using the softmax function, which performs a linear transformation on the third training word features. This is typically accomplished through a fully connected layer (or linear layer), which maps the third training word features to a new vector space. The dimension of this new vector is equal to the number of categories in the classification task. For a binary classification problem, the dimension of the output vector is 2. The linearly transformed new vector is then fed into the SoftMax function. The SoftMax function converts each element of the linearly transformed new vector into a probability value, such that the sum of all probabilities is 1. For a binary classification problem, SoftMax outputs two probability values, corresponding to the probabilities of the two categories, i.e., the probability that the current third training word feature is an entity vector. A value of 1 indicates that it is an entity vector, and a value of 0 indicates that it is not an entity vector.

[0055] 204. Calculate the cross-entropy loss by combining the classification result of whether the features of the third training word are entity vectors with the entity position labels of the third training text sentence;

[0056] In this embodiment of the invention, a loss function (e.g., cross-entropy loss) is calculated by comparing the entity location labels of the third training text statement with the classification results of whether the features of the third training word are entity vectors, and this function is used to update the model parameters.

[0057] 205. The first named entity recognition model is pre-trained using the cross-entropy loss.

[0058] In this embodiment of the invention, the model parameters are updated based on the gradient of the loss function to minimize the loss. This can be achieved through backpropagation algorithm and optimizer (such as SGD, Adam, etc.), thereby pre-training the first named entity recognition model so that the pre-trained first named entity recognition model can distinguish whether the word features of a text sentence are entity vectors.

[0059] For example, the loss L in the first stage is calculated by comparing the predicted results with the actual results. CD :

[0060]

[0061] Where p(x) i ) represents the prediction result for the i-th token, y i To represent its true result, y i=0 indicates that the i-th token is not an entity or part of an entity, otherwise it is an entity. L CD This loss will then be used to optimize the parameters of the first named entity recognition model.

[0062] 103. By labeling the first training text statement with entity vectors, a second training text statement is obtained; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement;

[0063] In this embodiment of the invention, through steps 101 and 102, it is possible to determine the part of the training text statement with entity vectors from the first training text statement obtained in step 101. We take this part of the training text statement as the second training text statement.

[0064] It should be noted that, in this embodiment of the invention, the processing of the first named entity recognition model through steps 101-103 can identify whether the features of a single training word are entity vectors. This can reduce the difficulty of the named entity recognition task, avoid the problem of redundant calculation of non-entity, and thus improve the overall accuracy of the named entity recognition model.

[0065] 104. The second training text statement is processed by the pre-trained second named entity recognition model to obtain the entity type of the second training text statement; the second named entity recognition model includes a second encoder and a softmax layer; the prototype vector of each second entity category label and the second training word features of the second training text statement are extracted by the second encoder; the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text statement;

[0066] In embodiments of the present invention, such as Figure 6 As shown, the process of obtaining the prototype vector of each second entity category label includes:

[0067] 401. Retrieve synonyms and / or subdomain words for the second entity category label using a large language model or dictionary;

[0068] In this embodiment of the invention, for each predefined entity, a pre-trained large language model, such as BERT, GPT series (such as GPT-3, GPT-4), or a dictionary is selected to retrieve synonyms or words in the subdomain.

[0069] 402. Construct a corresponding natural language word set for each of the second entity category tags and its synonyms and / or subdomain words;

[0070] In this embodiment of the invention, a set S of natural language words can be constructed for them. i ={w1, w2, ..., w n This is used to calculate more accurate and stable prototype vectors. For example, PER entity categories can be represented by words such as human (synonym) and John (subdomain term).

[0071] 403. Each word in the natural language word set is used as an expanded entity category label of the second entity category label;

[0072] In this embodiment of the invention, the set S of natural language words i ={w1, w2, ..., w n Each word in the} can represent an expanded solid category label.

[0073] 404. Pass the expanded entity category label of the second entity category label through the second encoder to obtain the prototype vector of the second entity category label.

[0074] In this embodiment of the invention, the second encoder in the second named entity recognition model can perform word segmentation on the expanded entity category label to extract the word embedding vector corresponding to each word, which is the prototype vector of the second entity category label.

[0075] For example, using a set of natural language data to compute a prototype vector for each label category, represented as:

[0076]

[0077] This invention uses a set of natural language words to replace the original labels, thereby initializing a more reliable and accurate feature space and prototype vectors. This greatly improves the model's learning and generalization capabilities in scenarios with few samples.

[0078] In a preferred embodiment of the present invention, in order to obtain a more accurate prototype vector, this embodiment also uses the entity category label of the target domain to adjust the prototype vector of the second entity category label, thereby constructing a prototype vector that is more consistent with the distribution of the target domain.

[0079] Among them, such as Figure 7 As shown, the process of obtaining the prototype vector of each second entity category label also includes:

[0080] 411. Obtain multiple fourth training text statements carrying fourth entity category labels; the fourth entity category labels are used to indicate the category of the fourth training word in the fourth training text statement;

[0081] 412. Pass the fourth training text statement through the second encoder to obtain the fourth training word features of the fourth training text statement;

[0082] 413. If the second entity category label is the same as the fourth entity category label, then the fourth training word feature is superimposed with the prototype vector of the second entity category label to obtain the updated prototype vector of each second entity category label.

[0083] For example, the prototype vector of each updated second entity category label is calculated as follows:

[0084]

[0085] in, Let λ1 be the prototype vector of the updated second entity category label for the j-th category, where λ1 is a hyperparameter, n represents the number of words in the natural language word set for the second entity category label, and w is the vector of the second entity category label. i The i-th word in the set of natural language words representing the second entity category label; Z represents the second encoder. j This refers to words in the fourth training text where the fourth entity category label of the j-th category is the same as the second entity category label, where z represents Z. j The word in the text.

[0086] In this embodiment of the invention, the calculation formula used to obtain the entity type of each second training word feature of the second training text sentence by calculating the distance between each second training word feature and the prototype vector of the corresponding second entity category label through the softmax layer is expressed as follows:

[0087]

[0088] Among them, y pred h represents the entity type of each feature of the second training word in the second training text statement. i This indicates that the second word is used to train word features. Let T represent the prototype vector of the updated second entity category label for the j-th category, and let T represent all categories.

[0089] 105. The two-level contrastive learning loss is calculated by using the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature.

[0090] In this embodiment of the invention, contrastive learning requires constructing positive and negative sample pairs. In each batch during training, there are multiple training data points. These training data points within the same batch are considered a set D. Data with the same label in the set are considered positive samples, and data with different labels are considered negative samples. After obtaining the positive and negative sample pairs, a two-level contrastive learning approach is designed based on token-to-token and token-to-prototype comparisons. Their formulas are as follows:

[0091]

[0092]

[0093] Among them, L t The comparative learning between tokens is also known as the comparative distance between the features of each second training word, h. + This indicates that in the current batch set D, there are tokens that are related to tokens. i Positive samples with the same label, h j Conversely. L p The contrastive learning between the token and its prototype vector is essentially the contrastive distance between the prototype vector of each entity category label and the corresponding second training word feature, p. i Indicates the prototype to which the current token belongs, p j This represents all other prototypes besides the one it belongs to. τ is a scalar temperature parameter used to control the sharpness of the softmax distribution. Si m This represents a global similarity metric. This metric method considers not only the similarity of individual words or phrases, but also their distribution and contextual relationships throughout the text. Common similarity metrics such as cosine similarity, Jaccard similarity coefficient, and Euclidean distance can be used.

[0094] 106. After adjusting the parameters of the named entity recognition model based on the two-level contrastive learning loss, continue training until the model training is completed, and obtain the trained second named entity recognition model.

[0095] In some preferred embodiments, distance loss can also be used to adjust the parameters of the named entity recognition model before continuing training; the distance loss is calculated by the absolute distance between the prototype vector of each entity category label and the corresponding second training word feature; the parameters of the second named entity recognition model are adjusted based on the two-level contrastive learning loss and the distance loss before continuing training until the model training is completed, and a trained named entity recognition model is obtained.

[0096] For example, in addition to the contrastive learning loss, the absolute distance between the token and the prototype also needs to be calculated and included in the final loss, as follows:

[0097]

[0098] d=(dis(h i ,p1),dis(h i p2), ..., dis(h i p n ))

[0099] In this embodiment of the invention, each loss function is used to quantify the degree of inconsistency between the model's prediction results and the true values. When the model's prediction performance is good, the loss value will be low; conversely, when the prediction performance is poor, the loss value will be high. Through the above-mentioned two-level contrastive loss function and distance loss function, the second named entity recognition model can be optimized and trained, thereby obtaining the trained second named entity recognition model.

[0100] For reference Figure 8 The illustrated two-level contrastive learning training device for a named entity recognition model includes:

[0101] The first acquisition module is used to acquire the first training text statement;

[0102] The first recognition module is used to process the first training text statement through a pre-trained first named entity recognition model to obtain a classification result of whether the features of the first training words are entity vectors; the first named entity recognition model includes a first encoder and a first classifier;

[0103] The first processing module is used to obtain a second training text statement by labeling the first training text statement with entity vectors; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement;

[0104] The second recognition module processes the second training text statement using a pre-trained named entity recognition model to obtain the entity type of the second training text statement. The named entity recognition model includes a second encoder and a softmax layer. The second encoder extracts the prototype vector of each second entity category label and the second training word features of the second training text statement. The softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text statement.

[0105] The second processing module is used to calculate the bi-level contrastive learning loss by using the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature; and to continue training the named entity recognition model after adjusting the parameters based on the bi-level contrastive learning loss until the model training is completed, thus obtaining the trained named entity recognition model.

[0106] ① This invention proposes a two-level contrastive learning based on prototype networks, which can more effectively capture and distinguish inter-class information, initialize more discriminative prototype vectors, and enhance the model's classification performance, robustness, and generalization in small sample scenarios.

[0107] ② The proposed two-stage model structure of the fusion prototype network in this invention reduces the difficulty of the model task while avoiding redundant computation.

[0108] ③ The method proposed in this invention, which enriches prototype vectors using natural language vocabulary sets and support sets, enriches the information sources of prototype vectors in scenarios with small samples, contributing to the calculation of more reliable and semantically meaningful prototype vectors.

[0109] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A two-level contrastive learning training method for a named entity recognition model, characterized in that, The method includes: Obtain the first training text statement; The first training text sentence is processed by the pre-trained first named entity recognition model to obtain the classification result of whether the features of the first training word are entity vectors; the first named entity recognition model includes a first encoder and a first classifier; A second training text statement is obtained by labeling the first training text statement with entity vectors; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement; The second training text statement is processed by a pre-trained second named entity recognition model to obtain the entity type of the second training text statement; the second named entity recognition model includes a second encoder and a softmax layer; the prototype vector of each second entity category label and the second training word features of the second training text statement are extracted by the second encoder; the distance between each second training word feature and the prototype vector of the corresponding second entity category label is calculated by the softmax layer to obtain the entity type of each second training word feature of the second training text statement; The bi-level contrastive learning loss is calculated by the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature. After adjusting the parameters of the named entity recognition model based on the two-level contrastive learning loss, the model continues to be trained until the training is completed, resulting in a second named entity recognition model that has been trained. The process of obtaining the prototype vector of each second entity category label includes: Use a large language model or dictionary to retrieve synonyms and / or subdomain words for the second entity category label; Each second entity category label and its synonyms and / or subdomain words are respectively constructed into a corresponding natural language word set; Each word in the natural language word set is used as an extended entity category label for the second entity category label; The expanded entity category labels of the second entity category label are passed through the second encoder to obtain the prototype vector of the second entity category label; The process of obtaining the prototype vector of each second entity category label also includes: Obtain multiple fourth training text statements carrying fourth entity category labels; the fourth entity category labels are used to indicate the category of the fourth training words in the fourth training text statements; The fourth training text statement is passed through the second encoder to obtain the fourth training word features of the fourth training text statement; If the second entity category label is the same as the fourth entity category label, then the fourth training word feature is superimposed on the prototype vector of the second entity category label to obtain the updated prototype vector of each second entity category label; The updated prototype vector for each second entity category label is calculated as follows: in, Let λ1 be the prototype vector of the updated second entity category label for the j-th category, where λ1 is a hyperparameter, n represents the number of words in the natural language word set for the second entity category label, and w is the vector of the second entity category label. i The i-th word in the set of natural language words representing the second entity category label; Z represents the second encoder. j This refers to words in the fourth training text where the fourth entity category label of the j-th category is the same as the second entity category label, where z represents Z. j Words in the text; The formula used to calculate the entity type of each second training word feature in the second training text sentence by calculating the distance between the prototype vector of each second training word feature and the corresponding second entity category label through the softmax layer is expressed as follows: Among them, y pred h represents the entity type of each feature of the second training word in the second training text statement. i Let T represent the features of the second word trained on it, and let T represent all categories.

2. The two-level contrastive learning training method for a named entity recognition model according to claim 1, characterized in that, The pre-training process of the first named entity recognition model includes: Obtain a third training text statement carrying entity location labels; the entity location labels are used to indicate whether the training words of the third training text statement are entity vectors. The third training word features of the third training text sentence are extracted by the first encoder to obtain the third training word feature set; The first classifier identifies the features of each training word in the third training word feature set, and the classification result of whether the third training word features are entity vectors is obtained. The cross-entropy loss is calculated by combining the classification result of whether the features of the third training word are entity vectors with the entity position labels of the third training text sentence. The first named entity recognition model is pre-trained using the cross-entropy loss.

3. The two-level contrastive learning training method for a named entity recognition model according to claim 1, characterized in that, The method further includes: The distance loss is calculated by the absolute distance between the prototype vector of each entity category label and the corresponding second training word feature; After adjusting the parameters of the second named entity recognition model based on the two-level contrastive learning loss and the distance loss, the model continues to be trained until the training is completed, resulting in a trained named entity recognition model.

4. A two-level contrastive learning training device for a named entity recognition model, characterized in that, The device includes: The first acquisition module is used to acquire the first training text statement; The first recognition module is used to process the first training text statement through a pre-trained first named entity recognition model to obtain a classification result of whether the features of the first training words are entity vectors; the first named entity recognition model includes a first encoder and a first classifier; The first processing module is used to obtain a second training text statement by labeling the first training text statement with entity vectors; the second training text statement has a second entity category label, which is used to indicate the category of the training words in the second training text statement; The second recognition module processes the second training text statement using a pre-trained named entity recognition model to obtain the entity type of the second training text statement. The named entity recognition model includes a second encoder and a softmax layer. The second encoder extracts the prototype vector of each second entity category label and the second training word features of the second training text statement. The softmax layer calculates the distance between each second training word feature and the prototype vector of the corresponding second entity category label to obtain the entity type of each second training word feature of the second training text statement. The second processing module is used to calculate the bi-level contrastive learning loss by using the contrast distance between the prototype vector of each entity category label and the corresponding second training word feature, as well as the contrast distance between each second training word feature; and to continue training the named entity recognition model after adjusting the parameters based on the bi-level contrastive learning loss until the model training is completed, thereby obtaining the trained named entity recognition model. The process of obtaining the prototype vector of each second entity category label includes: Use a large language model or dictionary to retrieve synonyms and / or subdomain words for the second entity category label; Each second entity category label and its synonyms and / or subdomain words are respectively constructed into a corresponding natural language word set; Each word in the natural language word set is used as an extended entity category label for the second entity category label; The expanded entity category labels of the second entity category label are passed through the second encoder to obtain the prototype vector of the second entity category label; The process of obtaining the prototype vector of each second entity category label also includes: Obtain multiple fourth training text statements carrying fourth entity category labels; the fourth entity category labels are used to indicate the category of the fourth training words in the fourth training text statements; The fourth training text statement is passed through the second encoder to obtain the fourth training word features of the fourth training text statement; If the second entity category label is the same as the fourth entity category label, then the fourth training word feature is superimposed on the prototype vector of the second entity category label to obtain the updated prototype vector of each second entity category label; The updated prototype vector for each second entity category label is calculated as follows: in, Let λ1 be the prototype vector of the updated second entity category label for the j-th category, where λ1 is a hyperparameter, n represents the number of words in the natural language word set for the second entity category label, and w is the vector of the second entity category label. i The i-th word in the set of natural language words representing the second entity category label; Z represents the second encoder. j This refers to words in the fourth training text where the fourth entity category label of the j-th category is the same as the second entity category label, where z represents Z. j Words in the text; The formula used to calculate the entity type of each second training word feature in the second training text sentence by calculating the distance between the prototype vector of each second training word feature and the corresponding second entity category label through the softmax layer is expressed as follows: Among them, y pred h represents the entity type of each feature of the second training word in the second training text statement. i Let T represent the features of the second word trained on it, and let T represent all categories.

Images