A method for coreference resolution based on multi-label sequence annotation

Abstract

The present invention provides a reference resolution method based on multi-label sequence annotation, comprising: S1, dividing a text to be processed into a plurality of text segments of preset length and non-overlapping; S2, using a pre-trained model to encode each text segment to obtain the semantic expression of each character; S3, judging whether each character belongs to a BIE label based on the semantic expression of each character; S4, based on the label corresponding to each character, in a continuous I label sequence, combining characters corresponding to any BE label based on a preset combination strategy to form a phrase, wherein the continuous I label sequence indicates that each character in a continuous character sequence has an I label; S5, calculating the similarity score between any two phrases, and retaining a preset number of phrases with a high similarity score ranking for each phrase as its preceding candidate phrases; S6, using a sorting model to sort the preceding candidate phrases of each phrase and using the preceding candidate phrase with the highest score as its mutual reference object.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and more specifically, to a coreference resolution technology in the field of natural language processing, and more specifically, to a coreference resolution method based on multi-label sequence annotation. Background Art

[0002] Co-reference resolution is an important part of natural language processing, providing key reference structure information for other technologies. Co-reference resolution aims to solve the problem of different phrases in the same document pointing to the same entity. The essence of the problem is to cluster different phrases pointing to the same entity into one cluster.

[0003] In order to achieve both convenience and effectiveness, end-to-end reference resolution technology has gradually become the main development direction in this field. Among them, the end-to-end reference resolution technology combines the phrase recognition module and the phrase mutual reference relationship judgment module and configures them into the model. The user inputs the original corpus without any special processing to the model, and the model directly outputs the mutual reference relationship between each phrase. It is not necessary to identify the phrases that need to be judged in advance, thereby eliminating the model's dependence on the phrase recognition module and indirectly reducing the error accumulation between different task modules. The existing end-to-end reference resolution technology is generally based on the phrase recognition module. This technology scores text fragments of all possible lengths by training a phrase ranking model, and retains the text fragments with the first λ (this is a hyperparameter, generally selected as 0.4, which represents the proportion of text fragments retained by the model, and this parameter plays the role of preliminary filtering of text fragments) as the input of the phrase mutual reference relationship judgment module. If the length of the input text is N, then the order of magnitude of the intermediate result is O(λN 2 ), where O is used to measure the time complexity of the algorithm. Therefore, limited by the length of the input text, the number of output phrases of this method is proportional to the square of the text length, which will produce more intermediate noise, resulting in lower overall performance of the model and making it difficult to meet the needs of industrialization. At the same time, the generation of too many intermediate results will also increase the noise in the second stage, thereby reducing the final effect of the model.

[0004] In addition, there are a large number of cross phrases in real corpus data. For example, in the text input "in Victoria Park, Hong Kong, which is located on the Kowloon Peninsula", "Hong Kong" and "Victoria Park, Hong Kong" belong to two different but intersecting phrase entities respectively. The existence of cross phrases makes the complexity of this task higher: for ordinary sequence labeling tasks, one character can only predict one label. For example, in the BIEO labeling method (the "B" label represents the start position of a phrase, the "I" label represents the middle position of a phrase, the "E" label represents the end position of a phrase, and the "O" label represents that this character does not belong to any phrase), for the character "Kong" in the sequence labeling task, predicting it as the "I" label or the "E" label, no matter which type is predicted, there will be at least one phrase that cannot be predicted correctly. Therefore, the conventional sequence labeling task is difficult to solve the problem of cross phrases. Summary of the Invention

[0005] Therefore, the object of the present invention is to overcome the above-mentioned defects of the prior art and provide a coreference resolution method based on multi-label sequence labeling.

[0006] A coreference resolution method based on multi-label sequence labeling according to the present invention is used to obtain the coreferential object of each phrase in the text to be processed. The method includes: S1, dividing the text to be processed into multiple non-overlapping text segments with a preset length; S2, encoding each text segment by using a pre-trained model to obtain the semantic expression of each character in each text segment; S3, respectively determining whether each character belongs to the BIE label based on the semantic expression of each character obtained in step S2; S4, based on the label corresponding to each character obtained in step S3, in the continuous I-label sequence, combining the characters corresponding to any BE label based on a preset combination strategy to form a phrase, where the continuous I-label sequence means that each character in a continuous character sequence has an I label; S5, calculating the similarity score between any two phrases, and retaining a preset number of phrases with the highest similarity scores for each phrase as its pre-candidate phrases; S6, using a ranking model to rank the pre-candidate phrases of each phrase and taking the pre-candidate phrase with the highest score as its coreferential object.

[0007] Preferably, the preset length is set to the maximum length supported by the pre-trained model. In some embodiments of the present invention, the pre-trained model is the SpanBERT model, and the preset length is 512.

[0008] Preferably, step S3 includes: using three binary classifiers to respectively determine whether each character belongs to the BIE label, wherein the three binary classifiers are a B label binary classifier, an I label binary classifier, and an E label binary classifier, respectively; the B label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character belongs to the B label as output; the I label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character is the I label as output; the E label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character is the E label as output.

[0009] Preferably, step S4 includes obtaining multiple continuous I label sequences for the label of each character in the text segment obtained in step S3, and performing the following steps for each continuous I label sequence: S41, constructing a phrase set and a phrase starting position set, and initializing the phrase starting position set to an empty set; S42, traversing the B label and E label of each character in the current continuous I label sequence, adding the character position with the B label to the phrase starting position set, and each time an E label is traversed, using each character position in the current phrase starting position set as the phrase starting position and the character position corresponding to the current E label as the ending position to generate multiple phrases and put them into the phrase set.

[0010] Preferably, step S5 includes using a bilinear attention mechanism to calculate the similarity score between any two phrases, and retaining a preset number of phrases with a higher similarity score than the phrase as its front candidate phrases, wherein the preset number is 15.

[0011] Compared with the prior art, the advantages of the present invention are: the performance optimization technology of end-to-end coreference resolution based on multi-label sequence annotation provided by the present invention gets rid of the model's dependence on the length of the article, improves the interpretability of phrase generation, reduces the noise of the system, and improves the overall performance of the model. At the same time, the multi-label sequence annotation technology can also solve the problem of potential cross-phrases, improving the final effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0013] Figure 1 A schematic diagram of a method for reference resolution based on multi-label sequence annotation according to an embodiment of the present invention;

[0014] Figure 2 FIG. 2 is a schematic diagram of an example according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0016] The inventors found that the existing end-to-end reference resolution technology often focuses on improving the effect, but ignores the performance problem, which makes it difficult to industrialize the actual model. The existing model generally recognizes phrases based on segmentation, which can be described as follows: for a text with an input length of N, for any position start and end, as long as start≤end and end-start+1≤max_length, the model considers [start,end] to be a potential phrase; for all potential phrases, the model will score all phrases and only retain the phrases with scores before λ (λ is generally 40%); for phrases after filtering, the model will calculate the semantic similarity between any two phrases, and retain a fixed number (generally 50) of phrases as candidate phrases for each phrase; for any phrase and the fixed number of candidate phrases it retains, and the fixed number of phrases are sorted, the first one in the sorting will be output as the co-referential object of this phrase to the outside of the system. It can be seen from the above process that the existing model's method of generating phrases is too rough: for a text of length N, the potential order of division is O(N 2 ), even if the model implements a certain pruning strategy, that is, only retaining the phrases with the highest scores, the order of magnitude of the overall phrases is O(λN 2). Therefore, the complexity of the model is proportional to the square of the document length. This type of model is limited by the high order of phrases generated. When the model predicts text, it often consumes a lot of time, resulting in a contradiction between the low efficiency of the model and the high performance requirements of the industry, and the model is difficult to truly play its due role. For example, for the text "I love China" with a length of 4, the model will divide it into 10 phrases such as "I", "love", "China", "country", "I love", "love China", "China", "I love China", "love China" and "I love China". According to the retention ratio of 40%, 4 phrases will be retained. However, in fact, this text only has two phrases, "I" and "China", so this method will generate more additional noise, which will not only interfere with the model effect, but also greatly drag down the overall speed of the model and affect the performance of the system. For other methods, the non-end-to-end model divides the phrase recognition module and the phrase mutual reference relationship judgment module into two independent modules. The two modules do not affect each other. Although the flexibility of the model is improved to a certain extent, the segmentation of the model will increase the error accumulation between the models, thereby reducing the effect between the models. From a global perspective, non-end-to-end models still require two modules: phrase recognition and phrase mutual reference judgment, and the overall time does not decrease with the segmentation of the model.

[0017] In addition, the inventors found that the defects in the prior art were caused by excessive noise in the intermediate results output by the phrase recognition module (because the number of output phrases is strongly coupled with the length of the text) when conducting end-to-end reference resolution performance optimization research. After relevant research on phrase recognition and named entity recognition, the inventors found that this defect can be solved by predicting the BIE label of each character (representing the start, middle and end marks of the phrase respectively) through a multi-label sequence labeling method, and obtaining phrases through a certain combination method to eliminate the dependence of the number of output phrases on the length of the text, thereby reducing system noise, improving the overall performance of the model, and solving the problem of potential cross-phrases. In short, based on the purpose of solving the above-mentioned problems, the present invention provides an end-to-end reference resolution method based on multi-label sequence labeling, which generates intermediate phrases through a multi-label sequence labeling method, so that the number of intermediate results is freed from the dependence on the length of the text, reduces the noise of the intermediate results, and improves the generation quality of the intermediate phrases, so as to solve the problem in the prior art that the overall performance of the model is difficult to meet the requirements of industrialization due to excessive intermediate noise.

[0018] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] According to one embodiment of the present invention, Figure 1 As shown, a method for reference resolution based on multi-label sequence annotation is provided, and the method includes steps S1, S2, S3, S4, S5, and S6. Each step is described in detail below.

[0020] In step S1, the text to be processed is dynamically segmented into multiple text segments of preset length and non-overlapping, and the integrity of each segment is maintained as much as possible. According to one embodiment of the present invention, the preset length is the maximum length supported by the pre-trained model used in the subsequent steps of the present invention, which is generally 512. The integrity of the segment refers to the segmentation unit with sentences as the segmentation unit, and the same sentence appears in only one segment as much as possible. In special cases, the segments can be divided by commas. For Chinese text, the segments are first divided according to ".", "!", "?" and "......", and the length of each segment is kept less than the maximum length as much as possible; if a segment cannot meet this restriction, that is, the length of a single sentence is greater than the maximum length, then the segments are divided according to "," and ";"; for English text, WordPiece segmentation is first required, and then the segmentation is performed according to the above steps. In the above steps, the two special characters "[CLS]" and "[SEP]" need to be calculated into the segment length. The purpose of non-overlapping segments is to reduce the number of model inputs while allowing the pre-trained model to process long texts.

[0021] In step S2, each text segment is encoded using a pre-trained model to obtain a semantic expression of each character in each text segment. The semantic expression of the character refers to the result generated by the last layer of encoder in the pre-trained model. According to one embodiment of the present invention, the pre-trained model uses SpanBERT.

[0022] In order to better understand the present invention, the following briefly introduces the relevant knowledge of the pre-training model.

[0023] The so-called pre-trained model is a model that is pre-trained on a large-scale unsupervised data set and can be used directly in the field of language processing. With the gradual development of technology, pre-trained models represented by BERT (Bidirectional Encoder Representation From Transformers) have gradually become the main development direction of language models. Pre-trained models can learn a large amount of general language knowledge through pre-training and make the model more adaptable to downstream tasks by training more general tasks on large-scale corpora and then fine-tuning them on specific downstream tasks. The pre-trained model has a strong generalization ability for downstream tasks, has great advantages for small-scale data, and can solve the difficulties of polysemy and synonymous words. Using this type of pre-trained model for the present invention can realize the encoding of text fragments, among which, pre-trained models represented by BERT, ELMo, and SpanB ERT models are all available. Thanks to BERT's self-attention mechanism (self-Attention) and multi-head attention mechanism (MultiHead Attention), the model has very good performance in many natural language processing tasks. Among them, BERT's Masked Language Model is not good at natural language processing tasks related to text fragments, including question answering and coreference resolution, because the granularity of random masking is at the character level. In order to make the pre-trained model more suitable for natural language processing tasks related to text fragments, the SpanBERT model has emerged. From the above introduction, we can see that the general pre-training models include MLM and NSP pre-training tasks. SpanBERT is based on BERT and modifies the MLM model to change token prediction to span prediction. The SpanBERT model proposes a masking strategy at the text fragment level: that is, randomly masking a continuous range of characters and predicting the masked characters. Secondly, in the coreference resolution task, the model uses the boundaries of the phrase to generate the expression of the phrase. For this reason, SpanBERT also proposes the SBO (Span Boundry Objective) prediction task: Let's assume that the output of the last layer of SpanBERT is (x1, x2, x3, x4,…, x N ), where the segment to be predicted is (x s ,x e ), x s 、x e Represent the start and end characters of the continuous segment respectively; SBO task uses x s-1 and x e+1, that is, the character before the start character and the character after the end character to generate a new expression for any character in the segment,

[0024] y i =f(x s-1 ,x e+1 ,p i-s+1 )

[0025] Among them, p i-s+1 Represents x i Distance x s The position vector of . The function f can be calculated using the following formula:

[0026] h0=[x s-1 ;x e+1 ;p i-s+1 ]

[0027] h1=LayerNorm(W0h0)

[0028] y i =LayerNorm(W1h1)

[0029] In the above formula, w0 and w1 are parameters that need to be learned, h0 and h1 represent intermediate vectors, and y i represents the expression of the i-th character generated using boundary characters.

[0030] During the model optimization process, the above-mentioned method of using boundary character expressions to generate new expressions of characters in the fragment can enable the model to use the context of the characters to predict the characters. In other words, fragment-level information will be stored on the boundary of the fragment, which is consistent with the way fragment expressions are generated in question-answering tasks and coreference resolution tasks.

[0031] After obtaining the new expression of any character in the segment, the model will use the expression to predict the character at that position. The specific loss function is as follows:

[0032] l(x i =a) = l MLM (x i =a)+l SBO (x i =a)

[0033] = -logP(a|x i )-logP(a|y i )

[0034] = -logP(a|x i )-logP(a|f(x s-1 ,x e+1 ,p i-s+1 ))

[0035] From the loss function, we can see that the loss function not only includes the loss function of SBO, but also the loss function of MLM (Maksed Language Model).

[0036] It can be seen that SpanBERT uses the same structure as BERT to cover a continuous segment, and through the pre-training task of SBO, the model can more effectively handle tasks related to the segment. The SpanBERT model can well understand the contextual information of the segment, accurately model the semantic information of the segment, and show strong generalization ability on different document sets. Based on the performance of SpanBERT, the present invention uses it as a pre-training model to obtain the semantic expression of each character in the text segment, and then uses multi-label sequence annotation to generate possible potential phrases, and finally uses the phrase mutual reference relationship judgment module to generate the final coreference relationship. It is generally believed that the bottom layer of the pre-training model often contains grammatical and semantic information that is irrelevant to the task, while the high-level layer often has semantic information that is highly relevant to the task. Therefore, using the output of the last layer of SpanBERT as the semantic expression of the character can make the character more effectively capture the contextual information, and is more conducive to the improvement of the text segment task effect. It should be noted that the pre-training model used in step S2 is not limited to the SpanBERT model, and any pre-training model that can meet the generation method requirements of the segment expression in the reference resolution task can be used, but relatively speaking, the SpanBERT model has the best effect.

[0037] In step S3, whether each character belongs to a BIE tag is determined based on the semantic expression of each character obtained in step S2; wherein a continuous I-tag sequence means that each character in a continuous character sequence has an I-tag.

[0038] According to one embodiment of the present invention, three binary classifiers are used to determine whether each character belongs to a BIE label, wherein the three binary classifiers are a B label binary classifier, an I label binary classifier, and an E label binary classifier, wherein the B label binary classifier is a binary classifier obtained by pre-training with the semantic expression of the character output by the pre-training model as input and whether the character belongs to the B label as output, the I label binary classifier is a binary classifier obtained by pre-training with the semantic expression of the character output by the pre-training model as input and whether the character is an I label as output, and the E label binary classifier is a binary classifier obtained by pre-training with the semantic expression of the character output by the pre-training model as input and whether the character is an E label as output. Due to the existence of potential cross-phrases, a character may belong to several different phrases, so a character may belong to different positions (BIE) in different phrases. By pre-training three binary classifiers, it is determined whether a character belongs to a BIE label. Specifically, we use the large version of SpanBERT (the vector embedding dimension is 1024, the number of heads in the multi-head mechanism is 16, and the number of encoder layers is 24. SpanBERT is a common pre-training model and will not be described in detail here) to obtain the expression of each character in the last layer: first, we combine the text fragments into the form of "[CLS]" + text content + "[SEP]", where the document content is the content after the tokenizer has been tokenized. Then, we input the combined text into SpanBERT to obtain the expression of each character. For any character x i , set up three binary classifiers, as follows:

[0039]

[0040] h B =W B x i

[0041] p(x i =B) = sigmoid(h B )

[0042] h I =W I x i

[0043] p(x i =I) = sigmoid(h I )

[0044] h E =W E x i

[0045] p(x i =E) = sigmoid(h E )

[0046] Among them, p(x i =B), p(x i =I) and p(x i =E) represent the character x i The probability of belonging to labels B, I, and E, W B , W I and W E They correspond to the learnable parameters of the three binary classifiers respectively, and the sigmoid() function is used as the activation function. In other words, the SpanBERT expression of each character is used to determine the label to which each character belongs. For any classifier, the model adds a fully connected layer (Full Connection Layer) to the character, and uses the sigmoid function as the activation function. For the B label classifier, if the output of the sigmoid function is greater than 0.5, the character is considered to belong to the B label; otherwise, the character is considered not to belong to this label, and the same applies to the classifiers of other category labels. The training of binary classifiers is a common technique and will not be described here. The probability of the BIE to which each position belongs is generated by a binary classifier. For the same character, it may have a B label and / or an E label at the same time as the I label, which simplifies the computational complexity of phrase generation, reduces intermediate noise, and solves the problem of potential cross-phrases.

[0047] In step S4, based on the label corresponding to each character obtained in step S3, in the continuous I label sequence, the characters corresponding to any BE label are combined based on the preset combination strategy to form a phrase; wherein, the continuous I label sequence means that each character in a continuous character sequence has an I label. The so-called preset combination strategy can be simply described as: when the I sequence is continuous, any BE label is combined to generate a new phrase. The so-called continuous I sequence is a continuous I label sequence.

[0048] According to an embodiment of the present invention, a plurality of continuous I-label sequences are obtained for the label of each character in the text segment obtained in step S3, and a phrase is generated for each continuous I-label sequence in the following manner: Initialize the phrase set phrase set and the starting position set of the phrase start set is an empty set; for a continuous sequence of I tags, traverse all B tags and E tags. If the category of the B tag is 1, add B-loc (the position of the character corresponding to the B tag) to the phrase start position set start set ; If the category of the E label is 1, record the position of the character corresponding to the E label as E-loc, and traverse the start setFor each B-loc, generate a phrase (B-loc, E-loc) and add it to the phrase set phrase set ; (B-loc, E-loc) represents a phrase consisting of characters from the position corresponding to B-loc in the continuous I tag sequence to the position corresponding to E-loc. If the traversal of this continuous I tag sequence ends, the starting position set of the phrase is reinitialized. set The set is empty, and the next continuous I label sequence is traversed. The above steps are repeated. If there is no continuous I label sequence, the process ends and the final result phrase is output. set . For each phrase in the set, two strategies are used to obtain the expression of the phrase: one is to use the maximum pooling method to extract the maximum number of each dimension in the phrase as the expression of the phrase; the other is to use the mean pooling method to add the expressions of all characters in the phrase together and then take the average as the final expression of the phrase. This dynamic combination of phrase generation can not only solve the problem of cross-phrases, but also maintain the maximum number of phrases generated, filter out phrases with low confidence, and ensure the recall rate of phrases. At the same time, BIE annotation makes the phrase generation process more explainable. Compared with the phrase generation strategy based on fragment selection, this method can greatly reduce the noise of the system and improve the performance of the model.

[0049] In step S5, the similarity score between any two phrases is calculated, and a preset number of phrases with a high similarity score are retained for each phrase as the preceding candidate phrases. According to one embodiment of the present invention, a bilinear attention mechanism is used to calculate the similarity score between any two phrases, and a preset number of phrases with a high similarity score are retained for each phrase as the preceding candidate phrases. The calculation formula of the bilinear attention mechanism (Bilinear Attention Network) in step 5 is as follows:

[0050]

[0051] Where S c (i,j) is the similarity score between phrase i and phrase j, g i and g j They represent the vectorized semantic expressions of phrase i and phrase j respectively, W represents a learnable parameter, and the bilinear attention mechanism is a common technology, which will not be elaborated here.

[0052] For any phrase i, after calculating the similarity score with all other phrases, all phrases are sorted in descending order according to the score, and the phrases with the top K scores are retained as the preceding candidate phrases of phrase i. The purpose of retaining the top K phrases with phrase similarity scores for each phrase is to reduce the number of candidate phrases and reduce the complexity of the model. K is generally equal to 15.

[0053] In step S6, the ranking model is used to sort the preceding candidate phrases of each phrase and the preceding candidate phrase with the highest score is used as its mutual reference object. The ranking model is a pre-trained model, which is a relatively common model in the field of language processing. Any model that can sort the preceding candidate phrases according to the weights of the preceding candidate phrases and the corresponding phrases can be used. The ranking model is not specifically limited or elaborated here. In the sorting process, the attention weight of the phrase and each of its preceding candidate phrases is first calculated, and the expression of each preceding candidate phrase is re-updated to obtain the global expression of each preceding candidate phrase, and then input into the ranking model for sorting; then, the ranking model sets an empty prefix for each phrase, which means that this phrase does not have a co-referential relationship with any other phrase, and the score of this empty prefix phrase will be fixed to 0, and does not participate in the ranking model. If the scores of the candidate phrases in the ranking model are all less than 0, then it is considered that this phrase does not have a co-referential object, that is, this phrase points to the empty prefix; otherwise, this phrase has a co-referential relationship with the phrase with the highest score. For example, for any phrase i, all its K candidate phrases are input into the ranking model. Assume that the first ranked phrase is j and its score is s. If s ≥ 0, then phrase j and phrase i have a co-reference relationship; otherwise, phrase i does not have a co-reference relationship with any other phrase.

[0054] The steps of the present invention are described in detail below with a specific example. Assuming the input text is "Hong Kong Victoria Garden is located on the Kowloon Peninsula, which is located in the south of Hong Kong City.", assuming that the maximum length supported by the pre-trained model is 512, K = 2:

[0055] According to step S1, the text segments are dynamically divided according to punctuation marks. Since the maximum input length of SpanBERT is limited to 512, this example is divided into one segment, that is, the text itself: "Hong Kong Victoria Garden is located on the Kowloon Peninsula, which is located in the south of Hong Kong City.".

[0056] According to step S2, based on step 2, input this example into the SpanBERT model, obtain the representations of all characters in the last layer, segment the input text, and add the "[CLS]" and "[SEP]" tokens to construct the input {"[CLS]", "Hong", "Kong", "Victoria", "Park", "is", "located", "in", "Kowloon", "Peninsula", ",", "it", "is", "located", "in", "the", "southern", "part", "of", "Hong", "Kong", "City", "."}, and finally output the representation of each character: denoted as x0 to x 25 .

[0057] According to step S3, use three binary classifiers to respectively judge whether these characters belong to the BIE tags. For example, for the B tag classifier, the second character "Hong" is judged as 1, meaning that this character may be the starting position of a certain phrase, and at the same time this character is also judged as an I tag, as Figure 2 shown, the total tag classification is as follows: "Hong" (the 1st character), "it" and "Hong" (the 19th character) are judged as B tags; "Hong", "Kong", "Victoria", "Park", "it", "Hong", "Kong" and "City" are judged as I tags; "Kong", "Park", "it" and "City" are judged as E tags.

[0058] According to step 4, generate phrases according to the dynamic strategy combination.

[0059] Take the first continuous I tag sequence "Hong Kong Victoria Park" in the example as an example. First, traverse the characters and add the positions where the B tags are located to start set , that is, start set = {1}; if an E tag is encountered, traverse start set , and generate phrases. The positions where the E tags are located are 2 and 8 respectively, that is, generate two phrases (1,2) and (1,8), corresponding to "Hong Kong" and "Hong Kong Victoria Park" respectively, and add this phrase to the set phrase set . Similarly, for other continuous I tag sequences, generate phrase sets according to this step. The final phrase set is phrase set = {(1,2),(1,8),(16,16),(19,21)}, corresponding to "Hong Kong", "Hong Kong Victoria Park", "it" and "Hong Kong City" respectively.

[0060] According to step S5, the similarity score between any two phrases is roughly calculated using the bilinear attention mechanism, and the top K pre-candidate phrases with the similarity score are retained for each phrase. Taking "Victoria Garden, Hong Kong" as an example, its similarity scores with "Hong Kong", "it" and "Hong Kong City" are 0.79, 1.12 and 0.34 respectively. After sorting in descending order, the top 2 phrases are retained as candidate phrases, that is, "Hong Kong" and "it"; similarly, for "Hong Kong", "it" and "Hong Kong City" are retained; for "it", "Hong Kong" and "Victoria Garden, Hong Kong" are retained; for "Hong Kong City", "it" and "Hong Kong" are retained.

[0061] According to step S6, the phrase and its preceding candidate phrase are sorted using a sorting model, and the phrase with the highest score is used as the mutual reference object of the phrase.

[0062] Taking "Victoria Garden, Hong Kong" as an example, its candidate phrases are "Hong Kong" and "it". The two retained phrases are input into the sorting model and sorted. After sorting, the scores of the two phrases are 0.89 and 2.11 respectively, so it is determined that "Victoria Garden, Hong Kong" and "it" have a co-reference relationship. Similarly, other co-reference relationships can be obtained. Finally, the final co-reference relationship set {"Victoria Garden, Hong Kong", "it"} and {"Hong Kong", "Hong Kong City"} are output.

[0063] In summary, the performance optimization technology of end-to-end coreference resolution based on multi-label sequence annotation provided by the present invention gets rid of the model's dependence on article length, improves the interpretability of phrase generation, reduces system noise, and improves the overall performance of the model. At the same time, the multi-label sequence annotation technology can also solve the problem of potential cross-phrases and improve the final effect of the model.

[0064] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0065] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0066] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0067] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for coreference resolution based on multi-label sequence annotation, used to obtain the mutual reference object of each phrase in the text to be processed, characterized in that: The method comprises: S1, dividing the text to be processed into a plurality of text segments of preset length and without overlap; S2, using the pre-trained model to encode each text segment to obtain the semantic expression of each character in each text segment; S3, judging whether each character belongs to a BIE tag based on the semantic expression of each character obtained in step S2; step S3 includes: Three binary classifiers are used to determine whether each character belongs to the BIE label, wherein the three binary classifiers are a B label binary classifier, an I label binary classifier, and an E label binary classifier. The B label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character belongs to the B label as output. The I label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character is the I label as output. The E label binary classifier is a binary classifier pre-trained with the semantic expression of the character output by the pre-trained model as input and whether the character is the E label as output. S4, based on the label corresponding to each character obtained in step S3, in the continuous I label sequence, the characters corresponding to any BE label are combined based on a preset combination strategy to form a phrase, wherein the continuous I label sequence means that each character in a continuous character sequence has an I label; step S4 includes obtaining a plurality of continuous I label sequences for the label of each character in the text segment obtained in step S3, and performing the following steps on each continuous I label sequence: S41, constructing a phrase set and a phrase start position set, and initializing the phrase start position set to an empty set; S42, traversing the B label and E label of each character in the current continuous I label sequence, adding the character position with the B label to the phrase start position set, and each time an E label is traversed, using each character position in the current phrase start position set as the phrase start position and the character position corresponding to the current E label as the end position to generate multiple phrases and put them into the phrase set; S5, calculating the similarity scores between any two phrases, and retaining a preset number of phrases with top similarity scores for each phrase as its preceding candidate phrases; S6. Use the ranking model to rank the preceding candidate phrases of each phrase and use the preceding candidate phrase with the highest score as its mutual reference object.

2. The method according to claim 1, characterized in that The preset length is set to the maximum length supported by the pre-trained model.

3. The method according to claim 2, characterized in that The pre-trained model is the SpanBERT model.

4. The method according to claim 3, characterized in that The preset length is 512.

5. The method according to claim 1, characterized in that The step S5 includes using a bilinear attention mechanism to calculate the similarity score between any two phrases, and retaining a preset number of phrases with top similarity scores for each phrase as its front candidate phrases.

6. The method according to claim 5, characterized in that The preset number is 15.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the steps of the method as claimed in any one of claims 1 to 6.

Images