Training methods, devices, and storage media for prototype networks

By generating entity type and boundary prototypes and optimizing the prototype network by combining sub-word vectors and position vectors, the problem of lack of entity boundary information in small sample named entity recognition is solved, thereby improving recognition accuracy and efficiency.

CN117371504BActive Publication Date: 2026-11-17RICOH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210750715.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-28
Publication Date
2026-11-17
Estimated Expiration
2042-06-28

AI Technical Summary

Technical Problem

Existing named entity recognition methods fail to effectively utilize entity boundary information in small sample scenarios, leading to severe overfitting and affecting the accuracy of classification results.

Method used

By generating entity type prototypes and entity boundary prototypes, and combining sub-word vectors and position vectors, the prototype network is optimized to incorporate entity boundary information. The network is trained using support sets and query sets, and the network parameters are optimized using stochastic gradient descent.

Benefits of technology

It improves the performance of named entity recognition, especially significantly enhancing the model's recognition accuracy in small sample scenarios. The boundary feature generation method is efficient and easy to implement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117371504B_ABST
    Figure CN117371504B_ABST
Patent Text Reader

Abstract

This application provides a training method, apparatus, and storage medium for a prototype network. The method includes: obtaining a training set; generating sub-word vectors and position vectors of sub-words in the training samples; generating a support set and a query set based on the training set; and training the prototype network using the support set and the query set. The training includes: inputting the sub-word vectors of the training samples in the support set into a first prototype network to generate entity type prototypes; calculating a first loss for the entity type prototypes using the query set; fusing the sub-word vectors and position vectors of the sub-words to obtain fused sub-word vectors; inputting the fused vectors of the training samples in the support set into a second prototype network to generate entity boundary prototypes; calculating a second loss for the entity boundary prototypes using the query set; obtaining a total loss based on the first loss and the second loss; and optimizing the first prototype network and the second prototype network based on the total loss. This application can improve the performance of prototype networks in named entity recognition tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of machine learning and natural language processing (NLP) technology, specifically to a training method, apparatus and storage medium for a prototype network for named entity recognition (NER). Background Technology

[0002] Named entity recognition (also known as entity identification, entity chunking, and entity extraction) is a fundamental task in Natural Language Processing (NLP), serving as a crucial tool for many NLP tasks such as information extraction, question answering systems, syntactic analysis, and machine translation. Named entity recognition aims to locate and classify named entities in text into predefined entity types, such as names of people, organizations, places, time expressions, quantities, currency values, percentages, etc. Therefore, named entity recognition is also a classification task.

[0003] In small-sample (few training samples) classification tasks, one of the most pressing issues to address is overfitting. Due to the limited number of training samples, general classification algorithms tend to overfit, leading to significant discrepancies between the classification results and the actual results. To mitigate the impact of overfitting caused by insufficient sample size, metric-based meta-learning methods, such as prototype networks, can be used.

[0004] Prototype networks were proposed to solve few-shot classification tasks. These networks project samples into a metric space where samples of the same class are close together, and samples of different classes are far apart. Figure 1 As shown, in this metric space, there are samples of three categories: C1, C2, and C3, and samples of the same category are relatively close to each other. To label an unlabeled sample x, sample x is projected onto this space, and the category to which x is closest is calculated, thus determining which category x belongs to.

[0005] Prototype networks were initially proposed to solve few-shot image classification tasks. Later, researchers transferred them to sentence classification tasks, but they are still rarely used for few-shot NER tasks. Existing methods for few-shot NER tasks use entity type information to construct entity type prototypes, which have achieved good results in few-shot entity classification tasks. However, strictly speaking, entity classification includes two sub-tasks: 1) determining the entity boundary; 2) determining the entity type. Existing methods only identify type information and ignore the entity boundary information. Summary of the Invention

[0006] At least one embodiment of this application provides a training method and apparatus for a prototype network, which can improve the performance of the prototype network in the named entity recognition task.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a method for training a prototype network, the prototype network being used for named entity recognition, the method comprising:

[0009] Obtain a training set, which includes multiple training samples;

[0010] Generate sub-word vectors and position vectors for sub-words in the training samples, and fuse the sub-word vectors and position vectors to obtain the fused vector of the sub-word;

[0011] A support set and a query set are generated based on the training set, and the prototype network is trained using the support set and the query set, wherein the support set and the query set are subsets of the training set, and the prototype network includes a first prototype network and a second prototype network, and the training includes:

[0012] The sub-word vectors of each sub-word of the support set training samples are input into the first prototype network to generate entity type prototypes, and the first loss of the entity type prototypes is calculated using the query set.

[0013] The fusion vector of each word of the support set training sample is input into the second prototype network to generate an entity boundary prototype, and the second loss of the entity boundary prototype is calculated using the query set.

[0014] The total loss is obtained based on the first loss and the second loss, and the first prototype network and the second prototype network are optimized based on the total loss.

[0015] Optionally, the method further includes:

[0016] The process is repeated in multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

[0017] Optionally, generating the position vectors of the subwords in the training samples includes:

[0018] The position information of the subword in the training sample is encoded to obtain the position vector of the subword.

[0019] Optionally, the position information of the subword in the training sample is: the absolute position of the subword in the training sample, or the relative distance between the subword and the subword at a specified position in the named entity where the subword is located.

[0020] Optionally, the sub-word vector and position vector of the sub-word are fused to obtain the fused vector of the sub-word, including:

[0021] The sub-word vector and position vector of the sub-word are concatenated to obtain the fused vector of the sub-word;

[0022] Alternatively, the word vector and position vector of the sub-word can be weighted and summed according to preset weights to obtain the fusion vector of the sub-word.

[0023] Optionally, the fused vectors of each word from the supporting training samples are input into the second prototype network to generate entity boundary prototypes, including:

[0024] Based on the position of each subword in the training sample within its corresponding named entity, the entity boundary type of each subword in the training sample is obtained.

[0025] The fusion vectors of each word in the support set training samples are input into the second prototype network to generate entity boundary feature vectors for each word.

[0026] The entity boundary feature vectors of the same entity boundary type in the support set are fused to obtain the entity boundary prototype corresponding to that entity boundary type.

[0027] Optionally, a second loss is calculated using the query set for the entity boundary prototype, including:

[0028] The fusion vectors of each sub-word in the training samples of the query set are input into the second prototype network to generate entity boundary feature vectors for each sub-word.

[0029] Based on the distance between the entity boundary feature vectors of each word in the training samples of the query set and the entity boundary prototypes of each word, the prediction results of the entity boundary prototypes of each word in the training samples of the query set are obtained.

[0030] The second loss of the entity boundary prototype is calculated based on the prediction results of the entity boundary types of each word in the training samples of the query set and the differences between the entity boundary types of each word in the training samples of the query set.

[0031] Optionally, optimizing the first prototype network and the second prototype network based on the total loss includes:

[0032] The stochastic gradient descent method is used to adjust the network parameters of the first prototype network and the second prototype network to reduce the total loss.

[0033] Optionally, the entity boundary prototype includes: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

[0034] Secondly, embodiments of this application provide a training apparatus for a prototype network used for named entity recognition, the training apparatus comprising:

[0035] A sample acquisition module is used to acquire a training set, which includes multiple training samples.

[0036] The vector generation module is used to generate sub-word vectors and position vectors of sub-words in the training samples, and to fuse the sub-word vectors and position vectors of the sub-words to obtain the fused vector of the sub-words;

[0037] A network training module is used to generate a support set and a query set based on the training set, and to train the prototype network using the support set and the query set. The support set and the query set are both subsets of the training set. The prototype network includes a first prototype network and a second prototype network. The training includes: inputting the sub-word vectors of each word from the training samples in the support set into the first prototype network to generate an entity type prototype; calculating a first loss for the entity type prototype using the query set; inputting the fusion vectors of each word from the training samples in the support set into the second prototype network to generate an entity boundary prototype; calculating a second loss for the entity boundary prototype using the query set; obtaining a total loss based on the first loss and the second loss; and optimizing the first prototype network and the second prototype network based on the total loss.

[0038] Optionally, the device further includes:

[0039] The iterative optimization module is used to repeatedly perform multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

[0040] Optionally, the vector generation module is further configured to encode the position information of the subword in the training sample to obtain the position vector of the subword.

[0041] Optionally, the network training module is further configured to obtain the entity boundary type of each sub-word in the training sample based on the position of each sub-word in its respective named entity; input the fusion vector of each sub-word in the support set training sample into the second prototype network to generate the entity boundary feature vector of each sub-word; and fuse the entity boundary feature vectors of the same entity boundary type in the support set to obtain the entity boundary prototype corresponding to the entity boundary type.

[0042] Optionally, the network training module is further configured to input the fusion vector of each sub-word of the training samples in the query set into the second prototype network to generate entity boundary feature vectors of each sub-word; obtain the prediction result of the entity boundary prototype of each sub-word of the training samples in the query set based on the distance between the entity boundary feature vectors of each sub-word of the training samples in the query set and each entity boundary prototype; and calculate the second loss of the entity boundary prototype based on the difference between the prediction result of the entity boundary type of each sub-word of the training samples in the query set and the entity boundary type of each sub-word of the training samples in the query set.

[0043] Optionally, the entity boundary prototype includes: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

[0044] Thirdly, embodiments of this application provide a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the method described above.

[0045] Compared with existing technologies, the prototype network training method and apparatus provided in this application learn and construct entity type prototypes and entity boundary prototypes based on entity type information and entity boundary information, thereby introducing entity boundary information into named entity recognition, which can improve the performance of named entity recognition, especially in scenarios with few sample named entity recognition, where it can more effectively improve model performance. In addition, the training method in this application uses an efficient and easy-to-implement boundary feature generation method. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 This is a schematic diagram illustrating the principle of a prototype network based on existing technology.

[0048] Figure 2This is a flowchart illustrating a training method for a prototype network according to an embodiment of this application.

[0049] Figure 3 This is an example diagram of the generated sub-word vectors in an embodiment of this application;

[0050] Figure 4 This is an example diagram of the feature vector for generating word positions in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of a training device for a prototype network according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of a training device for a prototype network according to another embodiment of this application. Detailed Implementation

[0053] To make the technical problems, technical solutions, and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments. In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and structures have been omitted.

[0054] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0055] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0056] Existing technologies, when using prototype networks for named entity recognition tasks, typically only use entity type information to construct entity type prototypes, without considering entity boundary information. This application's embodiments learn and construct entity type prototypes based on entity type information and entity boundary prototypes based on entity boundary information. This results in generated entity boundary prototypes containing more useful entity boundary information, thereby improving the performance of named entity recognition, especially in scenarios involving few-sample named entity recognition.

[0057] Please refer to Figure 2 This application provides a method for training a prototype network for named entity recognition. The method includes:

[0058] Step 21: Obtain the training set, which includes multiple training samples.

[0059] Here, the embodiments of this application obtain a training set, which includes multiple training samples labeled with named entity types. For ease of processing, in addition to the predefined named entity types, an additional entity type can be set during named entity labeling to label sub-words that do not belong to the predefined named entities. For example, in the training sample "Jobs founded NeXT Inc.with$7million", "Jobs" is labeled as a person's name, "NeXT Inc." is labeled as an organization's name, and "$7million" is labeled as a currency value. "founded" and "with" are both labeled as other entity types.

[0060] In actual annotation, the annotation can be based on the named entity type of the subword and the position of the subword in that named entity type. For example, “Jobs” can be annotated as “B-person name”, where B indicates that the subword is the first subword of a person’s name; “NeXT” and “Inc.” in “NeXTInc.” can be annotated as “B-organization name” and “E-organization name” respectively, where B indicates that NeXT is the first subword of the organization name and E indicates that Inc. is the last subword of the organization name; “$”, “7” and “million” in “$7million” can be annotated as “B-currency value”, “I-currency value” and “E-currency value” respectively, where B indicates that $ is the first subword of the currency value, I indicates that 7 is the middle subword of the currency value and E indicates that million is the last subword of the currency value.

[0061] Step 22: Generate the sub-word vector and position vector of the sub-word in the training sample, and fuse the sub-word vector and position vector of the sub-word to obtain the fused vector of the sub-word.

[0062] Here, in this embodiment of the application, natural language text is converted into feature vectors at the sub-word level. Sub-words refer to words in the vocabulary of a pre-trained language model, which are the units used by the pre-trained language model to encode natural language text. Some rare words are segmented in the vocabulary, such as "culmination" being segmented into {cu, ##lm, ##ination}, and are therefore called sub-words.

[0063] This application embodiment can input training samples into a pre-trained language model to obtain the sub-word vectors of each word in the training samples encoded by the pre-trained language model. The pre-trained language model is usually trained on a large dataset in advance, and the pre-trained language model includes, but is not limited to, any of the following models: BERT model, RoBERTa, T5, etc. Figure 3 This example illustrates how a pre-trained language model can be used to generate subword vectors from training samples. Here, X represents the input sentence (training sample); x i This represents a sub-word in the sentence (training sample). Figure 3 The value of i in the range is from 1 to 8; Indicates the final generated subword x i The sub-word vectors, where, Let θ represent a pre-trained language model, and θ0 represent the parameters of the pre-trained language model. These parameters can be fixed or continuously trained and updated in the method of this application embodiment. For ease of description, x1 is used here to represent the sub-word vector of a certain sub-word x.

[0064] When generating the position vectors of subwords in the training samples, this embodiment of the application can encode the position information of the subwords in the training samples to obtain the position vectors of the subwords. Specifically, the position information can be the absolute position of the subword in the training samples, or the relative distance (relative position) between the subword and a subword at a specified position in the named entity where the subword is located. The subword at the specified position can be the start or end subword in the named entity. The specific encoding algorithm can be a sine wave algorithm, a cosine wave algorithm, or a combination of both.

[0065] Taking the absolute positions of the sub-words as an example, in the training sample “Jobs founded NeXTInc.with$7million”, the absolute positions of the sub-words “Jobs”, “founded”, “NeXT”, “Inc.”, “with”, “$”, “7” and “million” are 1, 2, 3, 4, 5, 6, 7 and 8 respectively.

[0066] Taking the example of using the aforementioned relative position for the subword's position information, and the subword at the specified position being the starting subword in the named entity, the subword position information is converted into a feature vector (position vector) of the subword position through a positional encoding algorithm. Specifically, one way to calculate the relative position of a subword is as follows:

[0067] p = |p i -p s |

[0068] p ip is the absolute position of a word i in the training samples. s It is the absolute position of the starting word in the named entity containing word i within the training samples. Here, p i and p s All are the absolute positions of the subwords (e.g.) Figure 4 (The numbers 1, 2, 3, ... are used). Taking the sub-word "million" in "$7million" as an example, the corresponding p... i The value is 8. The named entity containing the subword "million" is "$7million", and the absolute position of its starting subword "$" is 6. Therefore, the p value corresponding to the subword "million" is calculated to be |8-6| = 2. If a subword does not belong to any predefined entity type (such as...), Figure 4 The term “founded” can be treated as another entity type, and its position vector can be generated in the same way.

[0069] One process for generating feature vectors (position vectors) of word positions using relative positional information is as follows: Figure 4 As shown. Where X represents the input sentence (training sample); x i This represents a sub-word in the sentence (training sample). Figure 4 The value of i in the range is from 1 to 8; Indicates the final generated subword x i The position vector, where, Indicates the subword x i Relative position information, Let δ0 represent a positional encoding algorithm, where δ0 represents the parameters of that algorithm. These parameters can be fixed or continuously trained and updated using the training method described in this application. For ease of description, x2 is used here to represent the position vector of a word x.

[0070] In addition, when fusing the sub-word vector and position vector of the sub-word, this embodiment of the application can concatenate the sub-word vector x1 and the position vector x2 of the sub-word to obtain the fused vector x3 of the sub-word; or, according to a preset weight, perform a weighted summation of the sub-word vector x1 and the position vector x2 of the sub-word to obtain the fused vector x3 of the sub-word.

[0071] Step 23: Generate a support set and a query set based on the training set, and train the prototype network using the support set and the query set. The support set and the query set are both subsets of the training set. The prototype network includes a first prototype network and a second prototype network. The training includes: inputting the sub-word vectors of each word from the training samples in the support set into the first prototype network to generate an entity type prototype; calculating a first loss for the entity type prototype using the query set; inputting the fusion vectors of each word from the training samples in the support set into the second prototype network to generate an entity boundary prototype; calculating a second loss for the entity boundary prototype using the query set; obtaining a total loss based on the first loss and the second loss; and optimizing the first prototype network and the second prototype network based on the total loss.

[0072] Here, in this embodiment of the application, a support set and a query set are generated based on the training set. For example, a portion of the training samples in the training set can be used as the support set, and the remaining training samples can be used as the query set, thereby using the support set and the query set to train and optimize the prototype network. For example, the training set D... L The samples were divided into a support set S and a query set Q in a 4:1 ratio.

[0073] In this embodiment, the prototype network includes a first prototype network for generating entity type prototypes and a second prototype network for generating entity boundary prototypes. The training process is described in detail below.

[0074] The generation process of the entity type prototype and the calculation of the first loss are as follows:

[0075] (1) Use the support set S to generate entity type prototype c.

[0076] Specifically, the sub-word vectors of each word in the support set training samples are input into the first prototype network to generate entity type feature vectors for each word. Then, the entity type feature vectors of the same named entity type in the support set are fused (e.g., by performing vector averaging) to obtain the entity type prototype corresponding to that named entity type. The number of entity type prototypes depends on the preset number of named entity types, such as person name prototypes, organization name prototypes, currency value prototypes, and other entity type prototypes. The above calculation process can be expressed by the following formula:

[0077]

[0078] Where m represents a named entity type; S m This indicates that all subwords belonging to the named entity type m in the support set are supported; |S m | indicates the number of all subwords belonging to the named entity type m in the support set; c mThis represents the entity type prototype corresponding to the named entity type m generated using the support set. x1 represents the sub-word vector of the sub-word x generated in step 22, i.e. This represents the first prototype network. This represents the network parameters of the first prototype network, which can be trained and updated in the method of this application embodiment. This represents the entity type feature vector generated after the sub-word vector x1 is input into the first prototype network.

[0079] (2) Use query set Q to calculate the loss (first loss) of entity type prototype c.

[0080] Specifically, the word vectors of each word in the training samples of the query set are input into the first prototype network to generate entity type feature vectors for each word. Then, based on the distance between the entity type feature vectors of each word in the training samples of the query set and the prototypes of each entity type, the predicted name entity type of each word in the training samples of the query set is obtained (e.g., the probability that the word belongs to each named entity type). Here, the distance calculation algorithm can be the L1 algorithm, cosine similarity calculation algorithm, etc. Then, based on the difference between the predicted name entity type of each word in the training samples of the query set and the name entity type of each word in the training samples of the query set, the loss of the entity type prototype (first loss) is calculated. Specifically, the loss of the entity type prototype (first loss) can be calculated using the cross-entropy loss function.

[0081] In this embodiment of the application, the generation process of the entity boundary prototype and the calculation of the second loss are as follows:

[0082] (1) Use the support set S to generate entity type prototype b.

[0083] In this embodiment of the application, the entity boundary types of a certain subword in the training sample include the following four types:

[0084] 1. Entity starting position: This indicates that the word is located in a named entity in the training sample, and that the word is the first word in that named entity.

[0085] 2. Entity end position: This indicates that the word is located in a named entity in the training sample and that the word is the last word in that named entity.

[0086] 3. Internal position of an entity: This indicates that the word is located in a named entity in the training sample, and the word is an intermediate word in the named entity. In this case, the word is not the first or last word in the named entity.

[0087] 4. External entity position indicates that the word is not in the named entities in the training sample, that is, the word is a word of other entity types.

[0088] For the four entity boundary types mentioned above, this application embodiment generates entity boundary prototypes corresponding to each entity boundary type, namely: entity start position prototype, entity end position prototype, entity internal position prototype, and entity external position prototype.

[0089] When generating entity type prototypes, firstly, based on the position of each word in the training samples within its corresponding named entity, the entity boundary type of each word in the training samples is obtained. Then, the fused vector x3 of each word from the training samples in the support set is input into the second prototype network to generate entity boundary feature vectors for each word. Next, the entity boundary feature vectors of the same entity boundary type in the support set are fused (e.g., by performing vector averaging) to obtain the entity boundary prototype corresponding to that entity boundary type. The above calculation process can be expressed by the following formula:

[0090]

[0091] Where n represents a certain entity boundary type, S n b represents all subwords that support the set of the entity's boundary prototype. n This represents the entity boundary type corresponding to the entity boundary type n generated using the support set. x3 represents the fusion vector of the sub-word x generated in step 22; This represents the second prototype network. This represents the network parameters of the second prototype network, which can be trained and updated in the method of this application embodiment. This represents the entity boundary feature vector of the subword x generated after the fusion vector x3 of the subword x is input into the second prototype network.

[0092] (2) Use query set Q to calculate the loss (second loss) of entity boundary prototype b.

[0093] Specifically, the fused vectors of each word in the training samples of the query set are input into the second prototype network to generate entity boundary feature vectors for each word. Then, based on the distance between the entity boundary feature vectors of each word in the training samples of the query set and the entity boundary prototypes, the prediction result of the entity boundary prototype of each word in the training samples of the query set (e.g., the probability that a word belongs to each entity boundary type) is obtained. Here, the distance calculation algorithm can employ the L1 algorithm, cosine similarity calculation algorithm, etc. Then, based on the difference between the prediction result of the entity boundary type of each word in the training samples of the query set and the entity boundary type of each word in the training samples of the query set, the second loss of the entity boundary prototype is calculated. For example, the loss of the entity boundary prototype (second loss) can be calculated using the cross-entropy loss function.

[0094] After obtaining the first loss and the second loss, embodiments of this application can derive the total loss based on the first and second losses. For example, the total loss can be obtained by weighted summation of the first and second losses according to preset weights. Then, the first prototype network and the second prototype network are optimized based on the total loss. During optimization, stochastic gradient descent is used to adjust the network parameters of the first and second prototype networks with the goal of reducing the total loss.

[0095] Through the above steps, this embodiment of the application combines entity type prototypes and entity boundary prototypes to introduce more auxiliary information (entity boundary feature information) into named entity recognition, thereby improving the performance of named entity recognition tasks. Especially in scenarios with few samples, this embodiment of the application can significantly improve recognition performance. Furthermore, the training method of this embodiment of the application, and the way boundary features are generated, are efficient and easy to implement.

[0096] Furthermore, embodiments of this application can repeat multiple rounds of iterative optimization until a preset termination condition is met, thereby obtaining the final prototype network. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network, and the set of query sets used in each round of iterative optimization covers all training samples in the training set.

[0097] For example, the training samples in the training set are divided into 5 parts, each containing multiple training samples. Then, in the first round of iterative optimization (epoch 1), the training samples from parts 1-4 are used as the support set, and the training samples from part 5 are used as the query set for one round of iterative training.

[0098] Then, based on the prototype network obtained from the first round of iterative optimization, a second round of iterative optimization (epoch2) is performed. In the second round of iterative optimization, the training samples from parts 2 to 5 are used as the support set and the training samples from part 1 are used as the query set for one round of iterative training.

[0099] Then, based on the prototype network obtained from the second round of iterative optimization, a third round of iterative optimization (epoch 3) is performed. In the third round of iterative optimization, the training samples from parts 1 and 3-5 are used as the support set, and the training samples from part 2 are used as the query set for one round of iterative training.

[0100] Then, based on the prototype network obtained from the third round of iterative optimization, a fourth round of iterative optimization (epoch4) is performed. In the fourth round of iterative optimization, the training samples from parts 1, 2, and 4-5 are used as the support set, and the training samples from part 3 are used as the query set for one round of iterative training.

[0101] Then, based on the prototype network obtained from the fourth round of iteration optimization, the fifth round of iteration optimization (epoch5) is carried out. In the fifth round of iteration optimization, the training samples from parts 1, 2, 3, and 5 are used as the support set, and the training samples from part 4 are used as the query set for one round of iteration training.

[0102] At this point, all training samples have been used as the query set for training. If the preset training termination conditions have been met (such as reaching the preset number of iterations or the preset convergence condition), training can end and the final prototype network can be output; otherwise, the next round of iteration optimization continues.

[0103] In each of the above iterations of optimization, the number of samples to be trained in each batch can be set, which will not be elaborated on in this article.

[0104] Based on the above methods, this application also provides an apparatus for implementing the above methods. Please refer to [link / reference]. Figure 5 This application provides a training apparatus for a prototype network used for named entity recognition. The training apparatus includes:

[0105] The sample acquisition module 51 is used to obtain a training set including multiple training samples;

[0106] The vector generation module 52 is used to generate sub-word vectors and position vectors of sub-words in the training samples, and to fuse the sub-word vectors and position vectors of the sub-words to obtain the fused vector of the sub-words;

[0107] The network training module 53 is used to generate a support set and a query set based on the training set, and to train the prototype network using the support set and the query set. The support set and the query set are both subsets of the training set. The prototype network includes a first prototype network and a second prototype network. The training includes: inputting the sub-word vectors of each word from the training samples in the support set into the first prototype network to generate an entity type prototype; calculating a first loss for the entity type prototype using the query set; inputting the fusion vectors of each word from the training samples in the support set into the second prototype network to generate an entity boundary prototype; calculating a second loss for the entity boundary prototype using the query set; obtaining a total loss based on the first loss and the second loss; and optimizing the first prototype network and the second prototype network based on the total loss.

[0108] By combining entity type prototypes and entity boundary prototypes with the above modules, this embodiment of the application introduces entity boundary feature information into named entity recognition, thereby improving the performance of named entity recognition tasks.

[0109] Optionally, the training device further includes the following modules ( Figure 5 (Not shown in the image):

[0110] The iterative optimization module is used to repeatedly perform multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

[0111] Optionally, the vector generation module is further configured to encode the position information of the subword in the training sample to obtain the position vector of the subword.

[0112] Optionally, the position information of the subword in the training sample is: the absolute position of the subword in the training sample, or the relative distance between the subword and the subword at a specified position in the named entity where the subword is located.

[0113] Optionally, the vector generation module is further configured to concatenate the sub-word vector and the position vector of the sub-word to obtain the fusion vector of the sub-word; or, according to preset weights, to perform a weighted summation of the sub-word vector and the position vector of the sub-word to obtain the fusion vector of the sub-word.

[0114] Optionally, the network training module is further configured to obtain the entity boundary type of each sub-word in the training sample based on the position of each sub-word in its respective named entity; input the fusion vector of each sub-word in the support set training sample into the second prototype network to generate the entity boundary feature vector of each sub-word; and fuse the entity boundary feature vectors of the same entity boundary type in the support set to obtain the entity boundary prototype corresponding to the entity boundary type.

[0115] Optionally, the network training module is further configured to input the fusion vector of each sub-word of the training samples in the query set into the second prototype network to generate entity boundary feature vectors of each sub-word; obtain the prediction result of the entity boundary prototype of each sub-word of the training samples in the query set based on the distance between the entity boundary feature vectors of each sub-word of the training samples in the query set and each entity boundary prototype; and calculate the second loss of the entity boundary prototype based on the difference between the prediction result of the entity boundary type of each sub-word of the training samples in the query set and the entity boundary type of each sub-word of the training samples in the query set.

[0116] Optionally, the entity boundary prototype includes: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

[0117] Optionally, the network training module is further configured to use stochastic gradient descent to adjust the network parameters of the first prototype network and the second prototype network to reduce the total loss.

[0118] Please refer to Figure 6 This application also provides a hardware structure block diagram of a prototype network training device, such as... Figure 6 As shown, the training device 600 includes:

[0119] Processor 602; and

[0120] Memory 604, wherein computer program instructions are stored.

[0121] When the computer program instructions are executed by the processor, the processor 602 performs the following steps:

[0122] Obtain a training set, which includes multiple training samples;

[0123] Generate sub-word vectors and position vectors for sub-words in the training samples, and fuse the sub-word vectors and position vectors to obtain the fused vector of the sub-word;

[0124] A support set and a query set are generated based on the training set, and the prototype network is trained using the support set and the query set, wherein the support set and the query set are subsets of the training set, and the prototype network includes a first prototype network and a second prototype network, and the training includes:

[0125] The sub-word vectors of each sub-word of the support set training samples are input into the first prototype network to generate entity type prototypes, and the first loss of the entity type prototypes is calculated using the query set.

[0126] The fusion vector of each word of the support set training sample is input into the second prototype network to generate an entity boundary prototype, and the second loss of the entity boundary prototype is calculated using the query set.

[0127] The total loss is obtained based on the first loss and the second loss, and the first prototype network and the second prototype network are optimized based on the total loss.

[0128] Furthermore, such as Figure 6 As shown, the model training device 600 also includes a network interface 601, an input device 603, a hard disk 605, and a display device 606.

[0129] The various interfaces and devices described above can be interconnected via a bus architecture. The bus architecture can include any number of interconnecting buses and bridges. Specifically, various circuits representing one or more central processing units (CPUs) and / or graphics processing units (GPUs), as represented by processor 602, and one or more memories, as represented by memory 604, are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is understood that the bus architecture is used to implement communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are well known in the art and therefore will not be described in detail herein.

[0130] The network interface 601 can be connected to a network (such as the Internet, a local area network, etc.), receive training samples and other data from the network, and save the received data to the hard disk 605.

[0131] The input device 603 can receive various instructions input by the operator and send them to the processor 602 for execution. The input device 603 may include a keyboard or a clicking device (e.g., a mouse, trackball, touchpad, or touchscreen).

[0132] The display device 606 can display the results obtained by the processor 602 executing instructions, such as displaying training progress.

[0133] The memory 604 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results and other data during the calculation process of the processor 602.

[0134] It is understood that the memory 604 in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. The memory 604 of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0135] In some implementations, memory 604 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 6041 and application program 6042.

[0136] The operating system 6041 includes various system programs, such as a framework layer, a core library layer, and a driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 6042 includes various applications, such as a browser, used to implement various application functions. Programs implementing the methods of the embodiments of this application can be included in application program 6042.

[0137] The methods disclosed in the above embodiments of this application can be applied to processor 602, or implemented by processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 602 or by instructions in the form of software. The processor 602 may be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 604. Processor 602 reads the information in memory 604 and, in conjunction with its hardware, completes the steps of the above method.

[0138] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0139] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0140] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0141] The process is repeated in multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

[0142] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0143] The position information of the subword in the training sample is encoded to obtain the position vector of the subword.

[0144] Optionally, the position information of the subword in the training sample is: the absolute position of the subword in the training sample, or the relative distance between the subword and the subword at a specified position in the named entity where the subword is located.

[0145] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0146] The sub-word vector and position vector of the sub-word are concatenated to obtain the fused vector of the sub-word;

[0147] Alternatively, the word vector and position vector of the sub-word can be weighted and summed according to preset weights to obtain the fusion vector of the sub-word.

[0148] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0149] Based on the position of each subword in the training sample within its corresponding named entity, the entity boundary type of each subword in the training sample is obtained.

[0150] The fusion vectors of each word in the support set training samples are input into the second prototype network to generate entity boundary feature vectors for each word.

[0151] The entity boundary feature vectors of the same entity boundary type in the support set are fused to obtain the entity boundary prototype corresponding to that entity boundary type.

[0152] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0153] The fusion vectors of each sub-word in the training samples of the query set are input into the second prototype network to generate entity boundary feature vectors for each sub-word.

[0154] Based on the distance between the entity boundary feature vectors of each word in the training samples of the query set and the entity boundary prototypes of each word, the prediction results of the entity boundary prototypes of each word in the training samples of the query set are obtained.

[0155] The second loss of the entity boundary prototype is calculated based on the prediction results of the entity boundary types of each word in the training samples of the query set and the differences between the entity boundary types of each word in the training samples of the query set.

[0156] Specifically, when the computer program is executed by the processor 602, it can also perform the following steps:

[0157] The stochastic gradient descent method is used to adjust the network parameters of the first prototype network and the second prototype network to reduce the total loss.

[0158] Optionally, the entity boundary prototype includes: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

[0159] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0160] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0161] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0163] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a prototype network, said prototype network being used for named entity recognition, characterized in that, The method includes: Obtain a training set, which includes multiple training samples; Generate sub-word vectors and position vectors for sub-words in the training samples, and fuse the sub-word vectors and position vectors to obtain the fused vector of the sub-word; A support set and a query set are generated based on the training set, and the prototype network is trained using the support set and the query set, wherein the support set and the query set are subsets of the training set, and the prototype network includes a first prototype network and a second prototype network, and the training includes: The sub-word vectors of each sub-word of the support set training samples are input into the first prototype network to generate entity type prototypes, and the first loss of the entity type prototypes is calculated using the query set. The fusion vector of each word of the support set training sample is input into the second prototype network to generate an entity boundary prototype, and the second loss of the entity boundary prototype is calculated using the query set. The total loss is obtained based on the first loss and the second loss, and the first prototype network and the second prototype network are optimized based on the total loss.

2. The method as described in claim 1, characterized in that, Also includes: The process is repeated in multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

3. The method as described in claim 1, characterized in that, Generating the position vectors of subwords in the training samples includes: The position information of the subword in the training sample is encoded to obtain the position vector of the subword.

4. The method as described in claim 3, characterized in that, The position information of the subword in the training sample is: the absolute position of the subword in the training sample, or the relative distance between the subword and the subword at a specified position in the named entity where the subword is located.

5. The method as described in claim 1, characterized in that, The sub-word vector and position vector of the sub-word are fused to obtain the fused vector of the sub-word, including: The sub-word vector and position vector of the sub-word are concatenated to obtain the fused vector of the sub-word; Alternatively, the word vector and position vector of the sub-word can be weighted and summed according to preset weights to obtain the fusion vector of the sub-word.

6. The method as described in claim 1, characterized in that, The fused vectors of each word from the support set of training samples are input into the second prototype network to generate entity boundary prototypes, including: Based on the position of each subword in the training sample within its corresponding named entity, the entity boundary type of each subword in the training sample is obtained. The fusion vectors of each word in the support set training samples are input into the second prototype network to generate entity boundary feature vectors for each word. The entity boundary feature vectors of the same entity boundary type in the support set are fused to obtain the entity boundary prototype corresponding to that entity boundary type.

7. The method as described in claim 6, characterized in that, The second loss of the entity boundary prototype is calculated using the query set, including: The fusion vectors of each sub-word in the training samples of the query set are input into the second prototype network to generate entity boundary feature vectors for each sub-word. Based on the distance between the entity boundary feature vectors of each word in the training samples of the query set and the entity boundary prototypes of each word, the prediction results of the entity boundary prototypes of each word in the training samples of the query set are obtained. The second loss of the entity boundary prototype is calculated based on the prediction results of the entity boundary types of each word in the training samples of the query set and the differences between the entity boundary types of each word in the training samples of the query set.

8. The method as described in claim 1, characterized in that, The optimization of the first and second prototype networks based on the total loss includes: The stochastic gradient descent method is used to adjust the network parameters of the first prototype network and the second prototype network to reduce the total loss.

9. The method as described in claim 1, characterized in that, The entity boundary prototypes include: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

10. A training apparatus for a prototype network, said prototype network being used for named entity recognition, characterized in that, The training device includes: A sample acquisition module is used to acquire a training set, which includes multiple training samples. The vector generation module is used to generate sub-word vectors and position vectors of sub-words in the training samples, and to fuse the sub-word vectors and position vectors of the sub-words to obtain the fused vector of the sub-words; A network training module is used to generate a support set and a query set based on the training set, and to train the prototype network using the support set and the query set. The support set and the query set are both subsets of the training set. The prototype network includes a first prototype network and a second prototype network. The training includes: inputting the sub-word vectors of each word from the training samples in the support set into the first prototype network to generate an entity type prototype; calculating a first loss for the entity type prototype using the query set; inputting the fusion vectors of each word from the training samples in the support set into the second prototype network to generate an entity boundary prototype; calculating a second loss for the entity boundary prototype using the query set; obtaining a total loss based on the first loss and the second loss; and optimizing the first prototype network and the second prototype network based on the total loss.

11. The apparatus as claimed in claim 10, characterized in that, Also includes: The iterative optimization module is used to repeatedly perform multiple rounds of iterative optimization until a preset termination condition is met. In each round of iterative optimization, different support sets and query sets are generated and used to train the prototype network. The set of query sets used in each round of iterative optimization covers all training samples in the training set.

12. The apparatus as claimed in claim 10, characterized in that, The vector generation module is also used to encode the position information of the subword in the training sample to obtain the position vector of the subword.

13. The apparatus as claimed in claim 10, characterized in that, The network training module is further configured to obtain the entity boundary type of each sub-word in the training sample based on the position of each sub-word in its respective named entity; input the fusion vector of each sub-word in the support set training sample into the second prototype network to generate the entity boundary feature vector of each sub-word; and fuse the entity boundary feature vectors of the same entity boundary type in the support set to obtain the entity boundary prototype corresponding to the entity boundary type.

14. The apparatus as claimed in claim 13, characterized in that, The network training module is further configured to input the fusion vector of each sub-word of the training samples in the query set into the second prototype network to generate entity boundary feature vectors of each sub-word; obtain the prediction result of the entity boundary prototype of each sub-word of the training samples in the query set based on the distance between the entity boundary feature vector of each sub-word of the training samples in the query set and each entity boundary prototype; and calculate the second loss of the entity boundary prototype based on the difference between the prediction result of the entity boundary type of each sub-word of the training samples in the query set and the entity boundary type of each sub-word of the training samples in the query set.

15. The apparatus as claimed in claim 10, characterized in that, The entity boundary prototypes include: entity start position prototype, entity end position prototype, entity interior position prototype, and entity exterior position prototype.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the prototype network as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Probabilistic graphical model-based text attribute extraction method and apparatus, computer device and storage medium

    WO2021135477A1

  • Named entity recognition method

    WO2022042125A1