Large model entity co-reference resolution method based on theme information enhancement

Abstract

The invention relates to a large model entity co-reference resolution method based on theme information enhancement. The method comprises the following steps: extracting all mentions and contexts of the mentions in an input document; the method comprises the following steps of: inputting a context of a mention into a TSECR model, obtaining a theme structure, splicing an input document, the theme structure and a prompt, obtaining and outputting a co-reference relationship among the mention, and linking the mention to a corresponding entity co-reference cluster, wherein the TSECR model is constructed based on a BERTopic model and a Transform with a prompt added. According to the method, the context theme structure information of the entity can be used as assistance, the identification capability of the model on the co-reference relationship is enhanced, and meanwhile, the calculation overhead is reduced.

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing and information extraction, and in particular to a large model entity coreference resolution method based on topic information enhancement. Background Art

[0002] Entity coreference resolution (ECR) is a core task in text processing and information extraction, which aims to identify whether mentions in a text refer to the same entity. ECR plays a key role in processing entity reference relationships, eliminating ambiguity, establishing semantic coherence, and improving the depth of semantic analysis. Effective solution of ECR ​​is of great significance for applications such as machine translation, information extraction, question-answering systems, and chatbots. It can not only improve the performance of these systems, but also achieve a higher level of natural language understanding and reasoning capabilities.

[0003] Previous entity coreference resolution methods based on LLMs have some key deficiencies. First, LLMs usually require extremely large computing resources for training and fine-tuning, which limits their application in resource-limited environments. Second, when dealing with specific domains or rare coreference patterns, these models often perform poorly due to the long tail problem of pre-training data. Currently, most LLMs use the Transformer architecture based on Self-Attention. The weights learned in its self-attention mechanism make the model tend to establish associations between high-frequency co-occurring words in the input sequence, while ignoring the associations between low-frequency co-occurring words. This bias causes the model to tend to assign common semantics to ambiguous or polysemous mentions, rather than accurately assigning the meaning of the specific entity it refers to based on the context. Long texts have longer reference spans and complex narrative structures, and the model often needs to truncate the document when processing, which further increases the difficulty of correctly identifying such complex coreference relationships. Therefore, although LLMs have performed well in many natural language processing tasks, there is still significant room for improvement in the specific task of coreference resolution, especially when dealing with long texts and complex contexts. In addition, the current LLMs-based methods have achieved more advanced performance mainly due to full parameter fine-tuning of models with larger parameters. Although this method can improve performance, it has the problems of high resource consumption and low efficiency. Full parameter fine-tuning requires a separate model to be trained for each downstream task, which increases the difficulty of deployment and maintenance. Summary of the invention

[0004] The purpose of the present invention is to provide a large model entity coreference resolution method based on topic information enhancement, which can use the contextual topic structure information of the entity as an aid to enhance the model's recognition ability of coreference relationships while reducing computational overhead.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A large model entity coreference resolution method based on topic information enhancement, comprising:

[0007] Extract all mentions and the context of said mentions in the input document;

[0008] The context of the mention is input into the TSECR model, the topic structure is obtained, the input document, the topic structure and the prompt are concatenated, the co-reference relationship between the mentions is obtained and output, and the mention is linked to the corresponding entity co-reference cluster, wherein the TSECR model is constructed based on the BERTopic model and the Transformer model with added prompts.

[0009] Optionally, the context of the mention is input into the TSECR model, and obtaining the topic structure includes: extracting the context of the mention using the BERTopic model to obtain the topic structure, wherein the BERTopic model is obtained by training a data set, and the data set is a document marked with a coreference relationship.

[0010] Optionally, adding the prompt to the Transformer model includes: adding the prompt to each layer of the Transformer model and randomly initializing it.

[0011] Optionally, concatenating the input document, the topic structure and the prompt includes:

[0012] In the embedding layer of the Transformer model, the input document is concatenated with the topic structure of the mention, and the hint of the embedding layer is added to obtain an input sequence;

[0013] The middle layer of the Transformer model splices the prompt of the middle layer before the input sequence to obtain an output result;

[0014] The output layer of the Transformer model maps the output results into labels to determine whether there is a co-referential relationship between the mentions.

[0015] Optionally, the intermediate layer of the Transformer model splicing the prompt of the intermediate layer before the input sequence includes: any layer of the intermediate layer splicing the prompt of the current layer before the input sequence of the previous layer to obtain the input sequence of any layer;

[0016] The input sequence of any layer is:

[0017] h (i+1) =TransformerLayer (i) ([P (i) ,h (i) ])

[0018] Among them, h (i+1) is the representation of the input sequence at the i+1th layer, h (i) is the input of the i-th layer, P (i) Hints added for each layer, TransformerLayer (i) is the i-th layer Transformer structure.

[0019] Optionally, linking the mention to a corresponding entity coreference cluster includes:

[0020] Determine whether there is a co-referential relationship between the mention and the entity;

[0021] If the mention refers to a stored entity, then include the mention in the entity coreference cluster of the stored entity;

[0022] If the mention does not co-reference the stored entity, a new entity co-reference cluster is created and cached.

[0023] Optionally, the entity coreference cluster is:

[0024]

[0025] Among them, P(e) is the coreference cluster, s(m,e) is the scoring function of whether the mention and entity e refer to each other, the set of e is E, s(m,e′) is the scoring function of whether the mention and entity e′ refer to each other, e′ refers to any entity in the document, and e refers specifically to the entity e that the current mention is comparing.

[0026] Optionally, obtaining the prompt includes: calculating a gradient of a loss function with respect to the prompt, and updating the gradient.

[0027] The beneficial effects of the present invention are:

[0028] 1) A method is proposed to enhance entity coreference resolution of pre-trained large models by using topic structure information from text context. This method makes full use of topic structure information extracted from text context and effectively alleviates the bias of existing methods in mention representation, thereby significantly improving the accuracy and robustness of the model in dealing with ambiguous or polysemous complex reference relations.

[0029] 2) A large-scale entity coreference resolution model (TSECR) based on topic structure information enhancement is designed. The topic structure extraction module of this model can automatically and accurately extract topic structure information closely related to the entity from the context; the coreference resolution module deeply integrates the acquired topic structure information with the prompt to achieve more accurate coreference relationship recognition.

[0030] 3) Extensive experimental verification was conducted on multiple public datasets. The results show that TSECR significantly outperforms the baseline model in performance, and compared with the latest pre-trained large model method, TSECR only needs about 1.1% of its parameters to achieve about 98% of its performance level, demonstrating the efficiency and practicality of this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A comparison of the effects of not adding topic structure information and adding topic structure information on the LLMs pre-training stage in an embodiment of the present invention;

[0033] Figure 2 This is a diagram of the entity coreference resolution model architecture based on topic information enhancement according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of extracting the subject structure of an embodiment of the present invention;

[0035] Figure 4 This is a prompt architecture diagram of an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of post-processing according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] After research, it was found that co-referential mentions refer to the same entity, and the same entity, when appearing in different positions, is often accompanied by similar topics. Therefore, the co-referential relationship between mentions is significantly correlated with the topic structure of its context. Using this correlation can alleviate the bias formed in the pre-training stage of LLMs, such as Figure 1 As shown in the figure. The topic structure information in the text reflects a more complex latent semantic relationship than the sentence structure, which can complement word embedding and jointly reveal the latent semantics in the text. Prompt-based learning is an efficient transfer learning method. During the training process, only about 0.1%-3% of the parameters in fine-tuning need to be updated to achieve the same accuracy as full parameter fine-tuning.

[0040] The present invention provides a large model entity coreference resolution method based on topic information enhancement, comprising:

[0041] Extract all mentions and the context of mentions in the input document;

[0042] The context of mention is input into the TSECR model to obtain the topic structure, concatenate the input document, topic structure and prompt, obtain the co-reference relationship between the output mentions, and link the mention to the corresponding entity co-reference cluster. The TSECR model is built based on the BERTopic model and the Transformer model with added prompts.

[0043] Furthermore, the context of mention is input into the TSECR model to obtain the topic structure, including: extracting the context of mention using the BERTopic model to obtain the topic structure, the BERTopic model is obtained by training a data set, and the data set is a document with a marked coreference relationship.

[0044] Furthermore, adding a prompt to the Transformer model includes: adding prompts to each layer of the Transformer model and randomly initializing them.

[0045] Furthermore, the input document, the topic structure and the prompt are concatenated including:

[0046] In the embedding layer of the Transformer model, the input document is concatenated with the topic structure of the mention and the hint of the embedding layer is added to obtain the input sequence.

[0047] The middle layer of the Transformer model splices the prompts of the middle layer before the input sequence to obtain the output result;

[0048] The output layer of the Transformer model maps the output results into labels to determine whether there is a co-reference relationship between mentions.

[0049] Furthermore, the intermediate layer of the Transformer model splices the hint of the intermediate layer before the input sequence, including: any layer of the intermediate layer splices the hint of the current layer before the input sequence of the previous layer to obtain the input sequence of any layer.

[0050] Furthermore, linking the mention to the corresponding entity coreference cluster includes:

[0051] Determine whether there is a co-reference relationship between mention and entity;

[0052] If the mention refers to a stored entity, include the mention in the entity coreference cluster of the stored entity;

[0053] If the mention does not coreference with a stored entity, a new entity coreference cluster is created and cached.

[0054] Furthermore, obtaining the prompt includes: calculating the gradient of the loss function with respect to the prompt, and updating it.

[0055] The present invention will be further described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0056] like Figure 2 As shown in FIG, the specific steps of the large model entity coreference resolution method based on topic information enhancement are as follows:

[0057] 1. Train BERTopic on the dataset to capture all kinds of topics.

[0058] 2. Scan the document, detect all mentions and extract their context. Input the mention context into the trained BERTopic model to assign a topic structure to it.

[0059] 3. Add hints to each layer of the Transformer model and initialize these hints randomly.

[0060] 4. Scan each mention from front to back one by one, and compare it with the cached entity coreference clusters one by one to determine whether there is a coreference relationship.

[0061] 5. Concatenate the input document with the mention and its topic structure. In each layer of the model, concatenate the mention to the front of the input sequence.

[0062] 6. Perform forward propagation to obtain the model results and link the mention to the corresponding entity coreference cluster.

[0063] The task of entity coreference resolution (ECR) is defined as follows: given a document as input word sequence X = {x 1 ,x 2 ,...,x n}, where n represents the length of the document. represents the number of all possible text spans in X. Let s i Represents the i-th span in X, in the range 1≤i≤N, with the starting index being FIRST(i) and the ending index being LAST(i). i ={x FIRST(i) ,x FIRST(i)+1 ,...,x LAST(i)-1 ,x LAST(i)}. If span s i A span that does not refer to an entity or any other entity is ignored, and all other spans are represented as m. j , m j Refers to any mention in the document, and 1≤j≤M≤N, where M is the number of all mentions in the document. Co-reference links between mentions define co-reference clusters and corresponding entities. Where m is the abbreviation of mention, and the set of e is E, e∈E. The purpose of coreference resolution is to learn a distribution P(·) over all entities e in the document for each mention, and finally cluster all mentions referring to the same entity into a coreference cluster e. That is:

[0064]

[0065] Among them, s(m,e) is the scoring function of whether the mention and entity e are co-referenced, and s(m,e′) is the scoring function of whether the mention and entity e′ are co-referenced. e′ refers to any entity in the document, while e refers specifically to the entity e that the current mention is comparing. Entity co-reference clusters are the specific manifestations of entities.

[0066] Mention extraction plays a vital role in coreference resolution, and high-quality mention extraction is a necessary prerequisite for hint-based coreference resolution. However, existing studies have shown that large models perform poorly in mention extraction tasks, but perform better in mention linking tasks. Therefore, in order to obtain better coreference resolution results, the SpanBERTlarge model proposed by Joshi et al. is used for mention extraction.

[0067] TSECR uses NTM to extract contextual topic structure information from the context of mention. NTM accepts the context of mention as input and outputs a set of topics related to the context of mention.

[0068]

[0069] In order to ensure that the extracted topic structure can accurately reflect the topic information of the mention context, this embodiment uses BERTopic in NTM to extract the topic structure of the mention context. The core idea is to use BERT to capture the semantic information of the document, convert the documents in the large-scale text dataset into the topic space, and automatically identify potential topics, and then use the topic modeling technology to cluster these semantic information. First, BERTopic needs to be trained on the dataset to capture all kinds of topics in the dataset, and then the mention context is passed as input to the trained model to assign it the topic structure of the mention context. The dataset used for training is usually in the CoNLL2012 format, which marks the words or phrases in the document and uses serial numbers to indicate the co-reference relationship. Figure 3 shown.

[0070] TSECR uses hints to integrate the contextual topic structure information of mentions into the word embedding vector space of mentions. Hints are an efficient parameter fine-tuning method. The principle is to add trainable additional parameters to the input layer or intermediate layer of the model, keep the parameters of the model itself fixed during the training process, and continuously optimize the hint parameters. Finally, the model can accurately understand the task objectives and deeply understand how the coreference relationship between mentions and their topic structure information interact with each other.

[0071] Assuming that the Transformer model has L layers, we need to add k hints in each layer of the Transformer model. These hints can be expressed as:

[0072] P (i) =[P 1(i) ,P 2 (i) ,...,P k (i) ] (3)

[0073] in, represents the representation of the jth prompt in the i-th layer, and d is the dimension of the word vector. The virtual word uses a random initialization method:

[0074]

[0075] In the embedding layer of the Transformer model, the document is concatenated with the topic structure of the two mentions, and then the embedding layer trainable hints are added. Assuming the input sequence length is n, the overall input sequence can be expressed as:

[0076] Input=[P,x 1 ,x 2 ,...,x n ] (5)

[0077] The input form of the embedding layer is:

[0078] "P 1 P 2 ...P k ,

[0079] <input document> {input document}< / input document> ,

[0080] <mention 1> {mention 1}< / mention 1> ,

[0081] <mention 2> {mention 2}< / mention 2> ,

[0082] <mention 1topic structure> {mention 1topic structure}< / mention 1topicstructure> ,

[0083] <mention 2topic structure> {mention 2topic structure}< / mention 2topicstructure> ,

[0084] Does{mention 1}refer to{mention 2}? "

[0085] Among them, {input document} represents the input document, {mention 1} represents the first mention, {mention 2} represents the second mention, {mention 1topic structure} represents the topic structure information of the first mention, and {mention 2topic structure} represents the topic structure information of the second mention. During the training process, for each hint, the optimal parameter value is found by maximizing its conditional probability on the coreference resolution task.

[0086] Additional hints are also added to the middle layers of the model. For a more complex task like coreference resolution, this not only adds more trainable task-specific parameters, improving the basic model's ability to fit the coreference resolution task, but also adds hints to deeper layers that can have a more direct impact on model predictions, making them more accurate and reliable. During the forward propagation process, the model encodes the input sequence through each layer. Assume that the input to the i-th layer is represented as h (i) , when each layer processes the input sequence, it will splice the prompt representation of the layer in front of it. After the transformation of the i-th layer, the output representation is:

[0087] h (i+1) =TransformerLayer (i) ([P (i) ,h (i) ])(6)

[0088] Among them, h (i+1) TransformerLayer is the representation of the input sequence at layer i+1. (i) is the i-th layer Transformer structure. In the output layer of the model, the output result is mapped to a label, where "yes" means there is a co-reference relationship between mention1 and mention2, mapped to 1; "no" means there is no co-reference relationship between the two, mapped to 0. Whether there is a co-reference relationship between mention and entity e is determined by the following formula:

[0089] s(m,e)=LLM θ (m,e) (7)

[0090] Here, θ is an additional trainable parameter added by the prompt.

[0091] The overall model architecture of the prompt is as follows Figure 4 As shown, the gray part represents the trainable parameters. During training, the gradient of the loss function with respect to the prompt parameters is calculated and updated, while the parameters of the base model itself are frozen.

[0092]

[0093] Where h is the learning rate, is the gradient of the loss function with respect to the prompt parameters.

[0094] In the post-processing stage, the binary classification results are organized into entity coreference clusters. Each mention is scanned from the beginning to the end of the document. For each scanned mention, it is compared with the cached entity coreference clusters one by one to determine whether there is a coreference relationship. In the comparison stage, the first nominal mention is selected from the entity coreference cluster and compared with the current mention. Figure 5 As shown, if the current mention refers to a stored entity e k , then include it in the coreference cluster:

[0095] e k =e k ∪{m i} (9)

[0096] If the current mention does not coreference with any stored entity e, a new entity coreference cluster is created and cached:

[0097] E=E∪{{m i}} (10)

[0098] This embodiment proposes a method to promote large-model ECR by using the contextual topic structure information of the text. In order to deal with the impact of ambiguous or polysemous complex reference relationships on the accuracy of coreference resolution, a neural topic model (NTM) is used to extract the contextual topic structure information of mention and integrate it into the word embedding space of the entity, enhancing the model's ability to identify and utilize the latent topic structure semantic information contained in the text. In order to solve the problem of high resource consumption and low efficiency caused by the full parameter fine-tuning large parameter model, the scheme adopts a parameter-efficient fine-tuning (PEFT) method based on prompt learning, and designs a prompt template to convert the ECR task into a binary question-answering task. Compared with traditional fine-tuning methods, prompts interact with the model in a more natural and intuitive way. During the training process, the parameters of the large model are frozen, and only a small number of additional prompt parameters are updated. Ultimately, prompts enable the model to accurately understand the task objectives and deeply understand how the coreference relationship between mentions interacts with its contextual topic structure information. In this way, the topic structure-enhanced entity coreference resolution (TSECR) model proposed in this embodiment can use the contextual topic structure information of the entity as an aid to enhance the model's ability to identify coreference relationships while reducing computational overhead. In particular, the model demonstrates strong support capabilities in long document scenarios.

[0099] In order to verify the effectiveness of the entity coreference resolution method based on topic structure enhancement, this example selects three widely used public datasets for experimental analysis, namely CoNLL2012, PreCo and LitBank. These datasets show significant differences in scale, document length, number of mentions and clusters, and domain diversity, which can test the effectiveness of the method proposed in this example.

[0100] Table 1 shows the detailed statistics of the three datasets, including the number of documents in the training set, development set, and test set, as well as key indicators such as the average number of words, number of mentions, and cluster size in the documents in each dataset. In addition, in scenarios where the dataset contains speaker metadata and speaker changes occur, this embodiment adds the speaker's name to the text to enhance the model's ability to recognize conversation dynamics.

[0101] Table 1

[0102]

[0103] This example uses the standard CoNLL-2012 evaluation index for evaluation. The evaluation index includes MUC, B 3 and CEAF f4 There are three sub-indicators, and the final evaluation indicator is the average of these three indicators. 3 and CEAF f4 The effects of coreference resolution are measured from three perspectives: link, mention, and entity. In addition, the LitBank dataset uses the official 10-fold cross-validation split and takes the average as the final result.

[0104] This example uses the large model Llama3-8B for experiments. This model has been instruction tuned, and its pre-training phase does not use the three datasets mentioned in this example, so it is suitable as the basic model for the prompts in this example. This example uses P-Tuning v2 to achieve efficient fine-tuning of prompt parameters. SpanBERT is used for mention extraction large The model has been fully trained on the OntoNotes dataset and has a high mention extraction accuracy, which can well complete the mention extraction task. When NTM is adapted on the dataset, in order to capture all the topics in the dataset in a more fine-grained manner, each data in the dataset is segmented and the length of the segmented sentences is set to 12. The prompt initialization length of each layer in the Llama3-8B model is set to 128. During the training process, the parameters of the model itself are frozen, and only the additional trainable parameters added by the prompt are updated. The total trainable parameters of the model are the sum of the prompt parameters of each layer of Llama3-8B. The learning rate is set to 0.02, the batch size is 4, and the maximum number of steps in the training process is set according to the different datasets: 5000 steps for CoNLL2012, 20000 steps for PreCo, and 250 steps for LitBank. Gradient accumulation is used to accelerate the training process during training.

[0105] The experimental results on three English datasets are shown in Tables 2 and 3. Table 2 is the experimental results table on the CoNLL2012 English dataset. The asterisk (*) indicates the use of additional training data; Predicted mentions indicates that mentions are predicted using a mention extractor; Gold mentions indicates that the standard answer mention is used directly, skipping the mention extraction step. Table 3 is the experimental results table on the PreCo and LitBank datasets. The results are presented in CoNLL F1 values; LitBank0 indicates the result of the 0th fold split in the ten-fold cross validation; the asterisk (*) indicates the use of additional training numbers. Overall, the TSECR model proposed in this embodiment significantly outperforms the baseline model in performance, and compared with the most advanced fine-tuning model, TSECR only needs to use about 1.1% of its parameters to achieve about 98% of its performance level. In addition, compared with the current non-fine-tuning method, the TSECR model shows the best performance. Under the Gold mentions condition, compared with multiple comparable models, the TSECR model also achieved the best performance.

[0106] Specifically, TSECR achieves suboptimal performance compared to small model supervised fine-tuning methods on the CoNLL2012 dataset. TSECR compares well to the baseline model longdoc-SpanBERT large , the performance is improved by 2.3%; on the PreCo and LitBank datasets, it is improved by 0.5% and 2.6% respectively. This is due to longdoc-SpanBERT large The model only relies on SpanBERT based on the Transformer architecture with self-attention mechanism large The model obtains the semantic representation of the mention, and the self-attention mechanism tends to establish an association between a word in the input sequence and the words with higher attention weights, ignoring those with lower attention weights. This bias prevents the model from fully considering the contextual information of the mention, which in turn causes the model to tend to assign common meanings to ambiguous and polysemous mentions rather than accurately assigning appropriate meanings based on the context. TSECR effectively improves longdoc-SpanBERT by explicitly extracting contextual topic structure information and using this information to assist the large model in identifying co-reference relationships. large It is worth noting that CorefQA achieves the best results among the small model supervised fine-tuning methods on the CoNLL2012 dataset, but this is mainly due to the improvement brought by the use of additional training data.

[0107] Comparing the supervised fine-tuning methods of large models on the CoNLL2012 dataset, TSECR can achieve comparable performance with SOTA using only a small fraction of its parameters. This further shows that by incorporating topic structure information into the word embedding space of entities and efficiently fine-tuning the parameters of large models, it is possible to significantly reduce computational overhead while maintaining high performance.

[0108] Among the non-fine-tuning methods compared on the CoNLL2012 dataset, TSECR outperforms coref-llm, a method that uses hard prompts for coreference resolution on GPT-4. Although the specific number of parameters of GPT-4 has not been made public, it is generally believed that its number of parameters has reached the trillion level, while ChatGLM3 used by TSECR has only 6B parameters. This shows that prompts are superior to hard prompts. Prompts can more effectively guide the model to accurately understand the task objectives and reduce the noise caused by artificially designed hard prompts, thereby improving model performance. In particular, the deep prompts used by TSECR enable the model to obtain prompt information at different network levels, providing a deeper guidance for the model, thereby improving the accuracy of the model.

[0109] On the CoNLL2012 dataset, additional experiments were conducted under the Gold mentions condition to eliminate the impact of mention extraction on model performance, so as to focus on testing the co-reference linking ability of the model. The results show that TSECR performs best in comparison with the other four methods. This shows that under the premise of high-quality mention extraction, the TSECR model proposed in this example shows a strong co-reference linking ability.

[0110] On both the PreCo and LitBank datasets, TSECR achieved suboptimal performance, while the amount of trainable parameters was only about 1.1% of the SOTA.

[0111] Overall, in the three datasets, the results of fine-tuning the large parameter model are generally better than the fine-tuning results of the small parameter model. This shows that the large model has a stronger representation ability than the small model due to its large number of parameters, and can capture complex data patterns more deeply and learn deeper knowledge. However, a larger number of parameters means higher training costs and computational overhead, which makes it particularly important and meaningful for TSECR to achieve high performance with only a small number of parameters.

[0112] In addition, we observe that TSECR achieves higher improvements on the LitBank dataset than on the OntoNotes or PreCo datasets. This may be because the documents in the LitBank dataset are generally longer, with longer reference spans and more complex narrative structures, making it more prone to ambiguous and polysemous complex reference phenomena. This suggests that in long document contexts, the contextual topic structure information of mentions can more effectively help large models identify complex reference phenomena. At the same time, the LitBank dataset has fewer training samples, and it is difficult for traditional fine-tuning models to fully learn the knowledge therein. The hint method used by TSECR provides the model with better flexibility and generalization, thereby performing better with fewer samples. Table 2 shows that compared with B 3 Compared with the evaluation indicators, TSECR has f4 The performance of the indicators is closer to SOTA. This may be because the three evaluation indicators focus on different angles. MUC and CEAF f4 It focuses on measuring the coreference resolution results from the perspective of links and entities, while B 3 This shows that TSECR is better at linking coreferences and identifying all entities than extracting all mentions.

[0113] Table 2

[0114]

[0115] Table 3

[0116]

[0117] In order to further verify the effectiveness of each module of the model in this embodiment, an ablation experiment was conducted on the CoNLL2012 dataset to test and analyze each module of the model. The ablation experiment results are shown in Table 4.

[0118] Table 4

[0119]

[0120] Here, "-" means removing the current module. -mention extraction means using dcoref, a rule-based unsupervised coreference resolution system developed by Stanford University, instead of SpanBERT. large For mention extraction, dcoref's mention extraction accuracy is low. The results show that F 1 The value dropped by 4.1%, indicating that mention extraction is crucial to the coreference model, proving that SpanBERT used in this example largeIt can effectively improve the accuracy of mention extraction, thereby ensuring the subsequent mention links. -Deep prompts refer to removing the prompts in the middle layer of the model and only retaining the prompts in the embedding layer, which is similar to prompt tuning. The results show that the effect is significantly reduced after removing the prompts in the middle layer, indicating that for complex tasks such as coreference resolution, the prompts in the embedding layer alone are not enough for the model to fully understand the task, proving the necessity of adding deep prompts. -BERTopic training refers to using BERTopic directly without training it on the dataset. As a result, the F1 value dropped by 1.4%, indicating that it is necessary to train BERTopic to learn the topic structure in a specific dataset. -Topic structure refers to deleting the contextual topic structure information of the mention in the prompt, and the F1 value dropped by 2.5, proving that adding contextual topic structure information can help the model correctly identify coreference relationships. In summary, the ablation experiment shows that the four modules have different degrees of positive effects on the overall model.

[0121] This example shows two cases from the CoNLL2012 dataset, which intuitively presents the topic structure in the document, as shown in Table 5. It can be observed that the context topic structure information of the mention helps to accurately obtain the semantics of the mention, especially when the two mentions are far apart, the topic structure information can cross the distance limit and help the model correctly identify the co-reference relationship between the mentions. In the first case, the distance between "it" and "The wedding" is far, and the baseline model only relies on the self-attention mechanism of the model. Due to the bias, "it" is regarded as a formal subject and not for reference, so it is impossible to link "it" with "The wedding". However, since the context topic structure of the word "The wedding" is almost the same as that of the word "it", this potentially indicates that the word "The wedding" and the word "it" may refer to the same entity. By comparing the context topic structure of the two mentions, TSECR accurately determines that there is a co-reference relationship between "it" and "The wedding". In the second case, "bank" is a polysemous word. "The bank" in the first part means bank, while "the bank" in the second part is ambiguous. Since "bank" is more commonly used as a bank, the baseline models are trained more on the bank meaning of the word "bank" during the pre-training stage, so these models tend to interpret "the bank" in the second part as a bank. However, TSECR accurately infers that the second "the bank" should refer to the river bank through the contextual topic structure of the mention, and thus correctly identifies that the two mentions do not have a co-reference relationship. Therefore, by using the contextual topic structure information of the mention as additional knowledge, it can not only help determine the specific meaning of the mention, but also establish a connection between the two mentions through the similarity of the contextual topic structure, thereby enhancing the model's ability to identify complex co-reference relationships.

[0122] Table 5

[0123]

[0124]

[0125] Experimental results on three English datasets show that TSECR improves the performance by 2.3%, 0.5%, and 2.6% respectively compared to the baseline model. In addition, compared to the current state-of-the-art model (SOTA), TSECR only requires about 1.1% of its parameters to achieve about 98% of its performance level. These results show that by deeply fusing topic structure information with prompts, the performance of large models in coreference resolution tasks can be effectively enhanced.

[0126] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A large model entity coreference resolution method based on topic information enhancement, characterized in that: include: Extract all mentions and the context of said mentions in the input document; The context of the mention is input into the TSECR model, the topic structure is obtained, the input document, the topic structure and the prompt are concatenated, the co-reference relationship between the mentions is obtained and output, and the mention is linked to the corresponding entity co-reference cluster, wherein the TSECR model is constructed based on the BERTopic model and the Transformer model with added prompts.

2. The large model entity coreference resolution method based on topic information enhancement according to claim 1 is characterized in that: Inputting the context of the mention into the TSECR model, and obtaining the topic structure includes: extracting the context of the mention using the BERTopic model to obtain the topic structure, wherein the BERTopic model is obtained by training a data set, and the data set is a document marked with a coreference relationship.

3. The large model entity coreference resolution method based on topic information enhancement according to claim 1 is characterized in that: The Transformer model to which the prompt is added includes: adding the prompt in each layer of the Transformer model and randomly initializing it.

4. The large model entity coreference resolution method based on topic information enhancement according to claim 1 is characterized in that: Joining the input document, the topic structure and the prompt comprises: In the embedding layer of the Transformer model, the input document is concatenated with the topic structure of the mention, and the hint of the embedding layer is added to obtain an input sequence; The middle layer of the Transformer model splices the prompt of the middle layer before the input sequence to obtain an output result; The output layer of the Transformer model maps the output results into labels to determine whether there is a co-referential relationship between the mentions.

5. The large model entity coreference resolution method based on topic information enhancement according to claim 4 is characterized in that: The middle layer of the Transformer model splices the hint of the middle layer before the input sequence, including: any layer of the middle layer splices the hint of the current layer before the input sequence of the previous layer, and obtains the input sequence of any layer; The input sequence of any layer is: h (i+1) =TransformerLayer (i) ([P (i) ,h (i) ]) Among them, h (i+1) is the representation of the input sequence at the i+1th layer, h (i) is the input of the i-th layer, P (i) Hints added for each layer, TransformerLayer (i) is the i-th layer Transformer structure.

6. The large model entity coreference resolution method based on topic information enhancement according to claim 1 is characterized in that: Linking the mention to the corresponding entity coreference cluster includes: Determine whether there is a co-referential relationship between the mention and the entity; If the mention refers to a stored entity, then include the mention in the entity coreference cluster of the stored entity; If the mention does not co-reference the stored entity, a new entity co-reference cluster is created and cached.

7. The method for resolving coreference of large model entities based on topic information enhancement according to claim 1 is characterized in that: The entity coreference cluster is: Among them, P(e) is the coreference cluster, s(m,e) is the scoring function of whether the mention and entity e refer to each other, the set of e is E, s(m,e′) is the scoring function of whether the mention and entity e′ refer to each other, e′ refers to any entity in the document, and e refers specifically to the entity e that the current mention is comparing.

8. The large model entity coreference resolution method based on topic information enhancement according to claim 1 is characterized in that: The acquisition of the prompt includes: calculating the gradient of the loss function with respect to the prompt, and updating the gradient.

Images