Text event relation extraction method and device
By constructing a sample set of multiple types of event relationship data and introducing bilateral matching losses, the LLM model is fine-tuned, which solves the problems of large training samples and poor model universality in the existing technology, and achieves high-accuracy multi-type event relationship extraction.
Patent Information
- Application Number
- CN202410917317.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art requires a large number of training samples in event relationship extraction, and the classification model is poor in versatility, so it is impossible to effectively extract multiple types of text event relationships. The model is sensitive to the arrangement order, which affects the extraction accuracy.
By constructing a sample set of text event relationship data of multiple types, fine-tuning training of the LLM model is carried out, bilateral matching losses are introduced to reduce the impact of permutation order, and event relationship extraction is transformed into question-and-answer tasks to improve the universality and accuracy of the model.
The number of training samples is significantly reduced, the model's ability to extract multiple types of event relationships is improved, the omission problem is avoided, and the extraction accuracy is improved.
Smart Images

Figure CN118917405A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular, to a method and device for extracting text event relations. Background Art
[0002] Event Relation Extraction is a task to identify the event relations between text events contained in unstructured text. Usually, the event relations between text events contained in the text may include multiple types of event relations. The task of event relation extraction can be further subdivided into multiple subtasks according to the type of event relations to be extracted.
[0003] For example, the relationship between text events contained in unstructured texts can usually include coreference relations, temporal relations, causal relations, sub-event relations, etc. The task of event relation extraction is usually divided into sub-tasks such as coreference relation extraction, temporal relation extraction, causal relation extraction, and sub-event relation extraction.
[0004] In practical applications, when performing event relationship extraction, each of the above-mentioned subtasks can be converted into a classification task, and then the relevant types of event relationships can be extracted from the text in a classified manner. For example, each type of event relationship to be extracted can be defined as a classification label to train a classification model separately, and then the classification model can be used to extract relevant types of event relationships from unstructured text. However, this method may require a large number of training samples. Summary of the invention
[0005] This specification proposes a method for extracting text event relations, the method comprising:
[0006] A data sample set for fine-tuning an LLM model is obtained; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text and having a text event relation of a target type with a designated text event; the target text event is a text event having a text event relation of a target type with the designated text event; the text event relation of the target type is a text event relation type designated from the multiple types of text event relations;
[0007] Inputting the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text;
[0008] Acquire the at least one target text event output by the LLM model, and calculate the error loss between the at least one target text event and the at least one annotated text event; and rearrange and combine the at least one target text event to obtain a plurality of text event sets, and perform bilateral matching on the plurality of text event sets with annotated text event sets composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculate the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set;
[0009] The error loss and the bilateral matching loss are used as model losses, and the model parameters included in the LLM model are adjusted to complete supervised fine-tuning training for the LLM model.
[0010] Optionally, the guiding prompt is a prompt text for describing the dependency relationship between text events contained in the text;
[0011] The method further comprises:
[0012] Performing dependency analysis on the text to obtain a dependency tree for describing dependency relationships between text events contained in the text;
[0013] The text events with dependency relationships contained in the text are extracted from the dependency tree, and prompt text for describing the dependency relationships between the text events contained in the text is constructed based on the extracted text events to serve as the guidance prompt corresponding to the text.
[0014] Optionally, extracting text event pairs with dependency relations contained in the text from the dependency tree includes:
[0015] A text event contained in the text and having a direct or indirect dependency relationship with the designated text event is extracted from the dependency tree.
[0016] Optionally, performing logical reasoning on the text based on the guidance prompt to select at least one target text event from text events contained in the text includes:
[0017] Based on the guidance prompt, logical reasoning is performed on the text, and at least one target text event having a text event relationship of the target type with the designated text event is selected from the text events included in the guidance prompt.
[0018] Optionally, before inputting the data samples in the data sample set as training samples into the LLM model, the method further includes:
[0019] Performing text event recognition on the data sample to determine the word-grams corresponding to the text event contained in the data sample;
[0020] A preset identifier is added at a position corresponding to the word element contained in the data sample; wherein the preset identifier is used to uniquely identify the text event contained in the data sample.
[0021] Optionally, the error loss is represented by a cross entropy loss between a probability distribution of at least one target text event and a probability distribution corresponding to the at least one annotated text event selected from text events contained in the text by performing logical reasoning on the text based on the guidance prompt.
[0022] Optionally, the error loss is represented by a weighted sum of a first cross entropy loss and a second cross entropy loss; wherein the first cross entropy loss is a cross entropy loss between a first probability distribution corresponding to the at least one target text event and a second probability distribution corresponding to the at least one annotated text event, by performing logical reasoning on the text based on the guidance prompt to select from the text events contained in the text; and the second cross entropy loss is a cross entropy loss between a third probability distribution corresponding to the at least one target text event and a second probability distribution corresponding to the at least one annotated text event.
[0023] Optionally, the bilateral matching loss is a loss constructed based on the probability value that a target text event included in the optimal text event set is the same text event as a target annotated text event included in the annotated text event set; wherein the target annotated text event is an annotated text event included in the annotated text event set and has the same position order as the target text event in the optimal text event set.
[0024] Optionally, the bilateral matching loss is expressed by the following formula:
[0025]
[0026] Among them, L bpm represents the bilateral matching loss; M represents the number of the at least one target text event selected from the text events contained in the text; θ(i) represents the i-th target text event contained in the optimal text event set; c i represents the i-th target annotated text event contained in the annotated text event set; p θ(i) (c i ) represents the probability that the i-th target text event included in the optimal text event set is the same text event as the i-th target annotated text event included in the annotated text event set.
[0027] Optionally, the at least one annotated text event contained in the data sample is represented by a type ID corresponding to the type of text event relationship between the at least one annotated text event and the specified text event; wherein the type ID is a type ID obtained by performing vocabulary mapping on the type based on a preset vocabulary.
[0028] Optionally, the data samples in the data sample set include positive samples and negative samples; wherein the positive samples refer to data samples in which the texts included contain text events having an event relationship with the specified text event of the specified target type; and the negative samples refer to data samples in which the texts included do not contain text events having an event relationship with the specified text event of the specified target type.
[0029] Before inputting the data samples in the data sample set as training samples into the LLM model, the method includes:
[0030] Randomly delete some negative samples contained in the data sample set; or,
[0031] Invalid negative samples in the negative samples included in the data sample set are identified, and the invalid negative samples identified from the data sample set are deleted; wherein the invalid negative samples are data samples in which no text event relationship of any form exists in the included text.
[0032] Optionally, identifying invalid negative samples among the negative samples included in the data sample set includes:
[0033] The data samples in the data sample set are input into the pre-trained LLM model so that the LLM model performs logical reasoning on the data samples to identify the invalid negative samples contained in the data sample set.
[0034] Optionally, before inputting the data samples in the data sample set as training samples into the LLM model, the step includes:
[0035] Performing text event recognition on the text contained in the positive sample in the data sample set to determine the word element contained in the text corresponding to the text event;
[0036] Replacing the remaining word-grams contained in the text other than the word-grams corresponding to the text event with word-grams having the same semantics as or similar semantics to the remaining word-grams, so as to generate an extended positive sample corresponding to the positive sample;
[0037] The extended positive samples are added to the data sample set to expand the number of positive samples in the data sample set.
[0038] Optionally, before inputting the data samples in the data sample set as training samples into the LLM model, the method includes:
[0039] Performing text event recognition on the text contained in the positive sample in the data sample set to determine the word element contained in the text corresponding to the text event;
[0040] Replacing the remaining word-grams contained in the text except the word-grams corresponding to the text event with mask identifiers, and inputting the positive sample after the replacement into the pre-trained LLM model, so that the LLM model performs logical reasoning on the positive sample and predicts the word-grams corresponding to the mask identifier;
[0041] Replacing the mask identifier with a word element corresponding to the mask identifier predicted by the LLM model to generate an extended positive sample corresponding to the positive sample;
[0042] The extended positive samples are added to the data sample set to expand the number of positive samples in the data sample set.
[0043] Optionally, the multiple types of event relationships include a combination of the following:
[0044] Co-reference relationship, temporal relationship, causal relationship, and father-child relationship.
[0045] Optionally, the text contained in the data sample is unstructured text.
[0046] Optionally, the text contained in the data sample is a document.
[0047] This specification also proposes a method for extracting text event relations, the method comprising:
[0048] Acquire a data sample for event relationship extraction; wherein the data sample comprises text and an instruction for instructing to extract at least one target text event from the text; the target text event is a text event having a target type of text event relationship with a specified text event; the target type of text event relationship is a text event relationship type specified from the multiple types of text event relationships;
[0049] Inputting the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects at least one target text event from the text events contained in the text, and completes event relationship extraction for the text;
[0050] The LLM model is an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose text events contained therein have the highest matching degree with the text events contained in the annotated text event set.
[0051] Optionally, the data sample further includes a guidance prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text;
[0052] Performing logical reasoning on the text and selecting at least one target text event from text events contained in the text includes:
[0053] Based on the guiding prompt, logical reasoning is performed on the text, and at least one target text event is selected from text events contained in the text.
[0054] Optionally, the data sample does not include a guidance prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text;
[0055] Performing logical reasoning on the text and selecting at least one target text event from text events contained in the text includes:
[0056] Logical reasoning is performed on the text to generate a guidance prompt for guiding the LLM model to select at least one target text event from the text events contained in the text, and logical reasoning is further performed on the text based on the guidance prompt to select at least one target text event from the text events contained in the text.
[0057] Optionally, the guiding prompt is a prompt text for describing the dependency relationship between text events contained in the text;
[0058] The generating of a guidance prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text includes:
[0059] Performing dependency analysis on the text to obtain a dependency tree for describing dependency relationships between text events contained in the text;
[0060] The text events with dependency relationships contained in the text are extracted from the dependency tree, and prompt text for describing the dependency relationships between the text events contained in the text is constructed based on the extracted text events to serve as the guidance prompt corresponding to the text.
[0061] Optionally, extracting text event pairs with dependency relations contained in the text from the dependency tree includes:
[0062] A text event contained in the text and having a direct or indirect dependency relationship with the designated text event is extracted from the dependency tree.
[0063] Optionally, further performing logical reasoning on the text in the guidance prompt to select at least one target text event from text events contained in the text includes:
[0064] Based on the guidance prompt, logical reasoning is further performed on the text, and at least one target text event having a text event relationship of the target type with the designated text event is selected from the text events included in the guidance prompt.
[0065] This specification also proposes a text event relationship extraction device, the device comprising:
[0066] A first acquisition module acquires a data sample set for fine-tuning the LLM model; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text and having a text event relation of a target type with a specified text event; the target text event is a text event having a text event relation of a target type with the specified text event; the text event relation of the target type is a text event relation type specified from the multiple types of text event relations;
[0067] A first input module inputs the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text;
[0068] A calculation module, which obtains the at least one target text event output by the LLM model, and calculates the error loss between the at least one target text event and the at least one annotated text event; and rearranges and combines the at least one target text event to obtain a plurality of text event sets, and performs bilateral matching on the plurality of text event sets with annotated text event sets composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculates the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set;
[0069] The training module uses the error loss and the bilateral matching loss as model losses, and performs parameter adjustment on model parameters included in the LLM model to complete supervised fine-tuning training for the LLM model.
[0070] This specification also proposes a text event relationship extraction device, the device comprising:
[0071] The second acquisition module acquires a data sample for event relationship extraction; wherein the data sample includes text and an instruction for instructing to extract at least one target text event from the text; the target text event is a text event having a text event relationship of a target type with a specified text event; the text event relationship of the target type is a text event relationship type specified from the multiple types of text event relationships;
[0072] A second input module inputs the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects at least one target text event from the text events contained in the text, and completes event relationship extraction for the text;
[0073] The LLM model is an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose text events contained therein have the highest matching degree with the text events contained in the annotated text event set.
[0074] In the above embodiment, on the one hand, by constructing a data sample set corresponding to multiple types of text event relations to be extracted, and fine-tuning the LLM model based on the data sample set, the LLM model can have a general extraction capability of extracting multiple types of text event relations from the text, without the need to perform model training for each type of text event relation separately, thereby significantly reducing the number of training samples required for fine-tuning the LLM model.
[0075] On the other hand, by converting the task of extracting text event relations having a specified target type with a specified text event from a text using LLM into a question-answering task of selecting at least one target text event relation having a specified target type with a specified text event from text events contained in the text using LLM, the LLM model can also have the ability to simultaneously extract multiple target text events having text event relations having a specified target type with a specified text event from the text, thereby avoiding the problem that, in the process of extracting text event relations, the LLM model cannot cover all types of text event relations to be extracted, and some text events to be extracted are omitted, thereby improving the accuracy of extracting text event relations from text using the LLM model.
[0076] Moreover, in the process of fine-tuning the LLM model, by introducing bilateral matching loss into the model loss, the influence of the arrangement order of at least one target text event selected by the LLM model from the text events contained in the text on the extraction result of the LLM model can be reduced, thereby further improving the accuracy of extracting text event relations from text using the LLM model. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0078] Figure 1 is a flow chart of a method for extracting text event relations shown in an embodiment of this specification;
[0079] Figure 2 is a schematic diagram of an instruction format shown in an embodiment of this specification;
[0080] Figure 3 is a schematic diagram of performing dependency analysis on a text shown in an embodiment of this specification;
[0081] Figure 4 is a schematic diagram of a data sample shown in an embodiment of this specification;
[0082] Figure 5 It is a framework diagram for fine-tuning training of an LLM model shown in an embodiment of this specification;
[0083] Figure 6 is a flow chart of a method for extracting text event relations shown in an embodiment of this specification;
[0084] Figure 7 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;
[0085] Figure 8 is a block diagram of a text event relationship extraction device shown in an embodiment of this specification;
[0086] Fig. 9 It is a block diagram of another text event relationship extraction device shown in an embodiment of this specification. DETAILED DESCRIPTION
[0087] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0088] Event Relation Extraction is a task to identify event relations between text events contained in unstructured text. The event relations between text events contained in the text usually include multiple types of event relations.
[0089] For example, the relationship between text events contained in unstructured text may generally include a coreference relation, a temporal relation, a causal relation, a sub-event relation, and the like.
[0090] In practical applications, the task of event relationship extraction can be further divided into multiple subtasks according to the types of event relationships that need to be extracted.
[0091] For example, the task of event relationship extraction can usually be divided into subtasks such as coreference relationship extraction, temporal relationship extraction, causal relationship extraction, and parent-child relationship extraction.
[0092] In the related art, when performing event relationship extraction, each of the above-mentioned subtasks can usually be converted into a classification task, and then the relevant types of event relationships can be extracted from the text in a classification manner;
[0093] For example, each type of event relationship that needs to be extracted can be defined as a classification label to train a classification model separately, and then the classification model is used to extract relevant types of event relationships from unstructured text.
[0094] However, this approach may require a large number of training samples. For example, in practical applications, for each of the subtasks mentioned above, a large number of training samples need to be prepared, and the number of training samples that need to be prepared may eventually reach O(n 2 ) level.
[0095] Moreover, classification models trained for different types of event relations can usually only be used to extract a certain type of event relation, and cannot be extended to the extraction of other types of event relations, which results in poor versatility of the trained classification models.
[0096] Based on this, this specification proposes a general event relationship extraction framework based on the LLM (Large Language Model) model that can support the extraction of various types of text event relationships from text.
[0097] During implementation, on the one hand, by constructing a data sample set corresponding to various types of text event relations to be extracted, and fine-tuning the LLM model based on the data sample set, the LLM model can have the general extraction capability of extracting various types of text event relations from the text.
[0098] In this way, it is no longer necessary to train models for each type of text event relationship, which can significantly reduce the number of training samples required for fine-tuning the LLM model. For example, the number of training samples required can be reduced from O(n 2 ) is reduced to O(n).
[0099] On the other hand, by converting the task of extracting a text event relationship having a specified target type with a specified text event from a text using LLM into a question-answering task of selecting at least one target text event relationship having a text event relationship having a specified target type with a specified text event from text events contained in the text using LLM, the LLM model can be enabled to simultaneously extract multiple target text events having a text event relationship having a specified target type with a specified text event from a text.
[0100] In this way, the problem of missing some text events to be extracted due to the LLM model failing to cover all types of text event relations to be extracted can be avoided during the process of extracting text event relations, thereby improving the accuracy of extracting text event relations from text using the LLM model.
[0101] Thirdly, during the fine-tuning training of the LLM model, bilateral matching loss can also be introduced into the model loss.
[0102] Since bilateral matching is a matching algorithm that does not need to consider the arrangement order; therefore, by introducing bilateral matching loss into the model loss, the influence of the arrangement order of at least one target text event selected by the LLM model from the text events contained in the text on the extraction results of the LLM model can be reduced, thereby further improving the accuracy of the LLM model in extracting text event relations from the text.
[0103] See also Figure 1 , Figure 1 The flowchart of a method for extracting text event relations shown in this specification includes the following execution process:
[0104] Step 102: Acquire a data sample set for fine-tuning the LLM model; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text that has a text event relation of a target type with a specified text event; the target text event is a text event that has a text event relation of a target type with the specified text event; the text event relation of the target type is a text event relation type specified from the multiple types of text event relations;
[0105] In the initial state, the above LLM model can specifically be a pre-trained LLM basic model.
[0106] For example, in practical applications, a pre-training method can be used to perform pre-training on the basis of a public text data set to create the above LLM model. It should be noted that the pre-training method used when training the LLM model is not specifically limited in this specification.
[0107] The above-mentioned text may specifically include any form of text that includes text events in the text content, which is not limited in this specification.
[0108] For example, the text may be an unstructured text containing text events. For example, in one example, the text may be an unstructured long text such as a document containing a large number of text event relationships.
[0109] It should be explained that the text events contained in the text, also known as event mentions, refer to the text fragments contained in the text that are used to describe events such as specific behaviors, specific activities, specific state changes, etc. A text event contained in the text is usually a semantically complete text unit, which can be composed of one or more tokens. These tokens can usually be used to describe the occurrence of an event and the components of an event.
[0110] For example, taking the above text as a news text, the above text event may specifically be a text segment related to some social hot events contained in the content of the news text.
[0111] The above-mentioned text event relationship contained in the text refers to the text fragments contained in the text that are used to describe the association or connection between different text events in the text. A text event relationship contained in a text is usually also a semantically complete text unit, and can also be composed of one or more word units, which can usually be used to describe the association or connection between text events contained in the text.
[0112] For example, in practical applications, the above text event relations may include coreference relations, temporal relations, causal relations, sub-event relations, and the like.
[0113] Among them, it is necessary to explain that when two text events contained in a text refer to the same target event in the real world, the two events are considered to have a co-referential relationship. When two text events contained in a text have a chronological relationship, the two events are considered to have a temporal relationship. The causal relationship reflects the relationship between text events in a sequential and cause-effect relationship. The parent-child relationship reflects the granularity and inclusion relationship between events.
[0114] In this specification, the task goal of extracting text event relations from text using the LLM model can specifically be to extract at least one target text event contained in the text that has a text event relation of a target type specified from multiple types of text event relations to be extracted with a specified text event.
[0115] Among them, the various types of text event relations to be extracted may include the above-mentioned coreference relations, temporal relations, causal relations, parent-child relations, etc. The above-mentioned specified text event refers to a basic text event specified from various text events contained in the text, from which a text event relation needs to be extracted.
[0116] The above-specified target type of text event relationship refers to a certain type of text event relationship to be extracted that is specified from the above-specified multiple types of text event relationships to be extracted.
[0117] For example, in actual applications, for the following text:
[0118] “Last year more than 3,0000civilians were Killed and another 4,500 injured in Afghanistan,with roughly a5% increase from 2010”
[0119] The text contains three types of text events, namely killed event, injured event and increase event. Assuming that the user needs to extract the parent event (parent event) of the killed event or the child event (child event) of the killed event contained in the text, the killed event contained in the text can be used as the text event specified above, and the event relationship type of parent-child event will be used as the text event relationship of the target type specified above.
[0120] In this specification, in order to enable the LLM model to have the ability to extract text event relations from the text, a pre-constructed data sample set can be obtained; wherein the data samples in the data sample set can correspond to multiple types of text event relations to be extracted; then, based on the data sample set, the pre-trained LLM model can continue to be fine-tuned.
[0121] In the initial state, the data samples in the above-mentioned data sample set may specifically include text, instructions for instructing to extract at least one target text event having a text event relationship of the above-mentioned specified target type with a specified text event from the text, and at least one annotated text event pre-annotated for the text.
[0122] The specific format of the above instructions is not particularly limited in this specification. In practical applications, different instruction formats may be used for different types of text event relations to be extracted.
[0123] For example, see Figure 2 Taking the above-mentioned preset multiple types of text event relationships, including coreference relationship, temporal relationship, causal relationship and parent-child relationship as examples, the instruction formats corresponding to the above-mentioned four types of text event relationships can be respectively as follows Figure 2 For example, Figure 2Taking the first instruction above as an example, for a coreference relation, the instruction may be "List the coreference event of XX", where XX represents the text event specified above, and the meaning of the instruction is to list the coreference events of the XX event.
[0124] Furthermore, since the task goal of extracting text event relations from text using the LLM model is to extract at least one target text event contained in the text that has a text event relation of a specified target type with a specified text event, the above-mentioned annotated text event specifically refers to a text event contained in the text that has a text event relation of the specified target type with the above-mentioned specified text event. The above-mentioned at least one annotated text event will be used as a label item corresponding to the text contained in the data sample.
[0125] It should be noted that the number of the at least one annotated text event depends specifically on the total number of text events actually contained in the text that have the text event relationship of the specified target type with the specified text event. In practical applications, in order to enable the LLM model to have the ability to extract all text events contained in the text that have the text event relationship of the specified target type with the specified text event, when annotating the texts contained in the data samples in the data sample set, all text events contained in these texts that have the text event relationship of the specified target type with the specified text event can be annotated.
[0126] In this specification, in order to enable the LLM model to have a general extraction capability of extracting multiple types of text event relations from text, the data samples in the above data sample set may specifically correspond to multiple types of text event relations to be extracted. For example, multiple types of text event relations may exist between the text events contained in the text in the above data sample set.
[0127] In this way, the types of text event relations corresponding to the texts contained in the data samples in the above data sample set can cover the multiple types of text event relations that need to be extracted, so that the LLM model completed by fine-tuning training can have the ability to extract each type of text event relations among the above multiple types of text event relations.
[0128] For example, in one example, the text in the above data sample set can contain multiple event relationship types such as co-reference relationship, temporal relationship, causal relationship, parent-child relationship, etc. In this way, it can be ensured that the LLM model completed by fine-tuning training has the extraction capability of each event relationship type mentioned above.
[0129] In practical applications, some unstructured texts, such as long texts such as documents, usually contain a large number of text event relations. When extracting text event relations with a specified text event having the above specified target type from such text, there may be multiple text events in the text events contained in the text that have text event relations with the above specified text event having the above specified target type.
[0130] In order to accurately extract all text events contained in the text that have the text event relationship of the specified target type with the above-mentioned specified text event, in the present specification, the above-mentioned LLM model can be fine-tuned and trained based on the above-mentioned data sample set, and the task of extracting text event relationships of the specified target type with the specified text event from the text using LLM can be converted into a MAQ (multiple answers question) task of selecting at least one target text event that has a text event relationship of the specified target type with the specified text event from the text events contained in the text using LLM.
[0131] Among them, the so-called MAQ task is a question-answering task that is different from the traditional MCQ (Multiple Choice Question) task. The MCQ task usually focuses on selecting the best answer from the candidate answers (such as the answer with the highest probability of being the correct answer). The MAQ task is different. The MAQ task focuses on selecting multiple correct answers from the candidate answers. It is not difficult to see that the MAQ task is more suitable for solving complex question-answering scenarios where the candidate answers contain multiple correct answers.
[0132] Accordingly, in order to allow the LLM model to have sufficient reference information in the process of reasoning based on data samples, the data samples in the above-mentioned data sample set may include, in addition to text for indicating instructions for extracting from the text at least one target text event that has a text event relationship of a target type specified from the multiple types of text event relationships with a specified text event, and at least one annotated text event pre-annotated for the text, in actual applications, a guiding prompt for guiding the LLM model to select the above-mentioned at least one target text event from the text events contained in the text.
[0133] It should be noted that the above guidance prompts may specifically encapsulate structured knowledge for guiding the LLM model to select the above at least one target text event from the text events contained in the text. The specific form of these structured knowledge is not particularly limited in this specification.
[0134] In one embodiment shown, the structured knowledge encapsulated in the guidance prompt may be dependency relations between text events contained in the text. Accordingly, the guidance prompt may be a prompt text for describing the dependency relations between text events contained in the text.
[0135] For example, in actual applications, dependency parsing can be performed on the text contained in the data samples in the above data sample set to obtain a dependency tree for describing the dependency relationship between text events contained in the above text. Then, text events with dependency relationships (such as text event pairs with dependency relationships) contained in the text can be extracted from the dependency tree, and prompt text for describing the dependency relationship between text events contained in the text can be constructed based on the extracted text events as a guidance prompt corresponding to the above text.
[0136] It should be explained that text dependency is a concept in the field of natural language processing. Its main goal is to determine the dependency between the words contained in the text and form a grammatical structure based on the determined dependency between the words contained in the text. Through dependency analysis, the deep grammatical and semantic structure of the text can be effectively revealed, providing support for advanced text analysis and understanding. Among them, in practical applications, when performing dependency analysis on the text, a main word (usually a verb) can usually be identified as the core, and other words can be connected to this core word through a specific type of connection relationship. These connection relationships express the grammatical and semantic structure within the sentence, such as subject, object, attributive, adverbial, etc.
[0137] For example, see Figure 3 , in an example, still taking the following text as an example:
[0138] “Last year more than 3,0000civilians were Killed and another 4,500 injured in Afghanistan,with roughly a5% increase from 2010”
[0139] The results of dependency analysis on the above text can be seen as follows: Figure 3 .
[0140] like Figure 3As shown in the figure, after dependency analysis of the above text, symbolic tags that reflect the dependency relationship between the word units will be added between the word units in the above text. These symbolic tags are usually divided into two categories, namely part-of-speech tags and dependency relationship tags. These tags help to indicate the grammatical function of words and the structural relationship between them. Among them, it needs to be explained that Figure 3 The part-of-speech tags shown in are VBN, CC, DT, CD, IN, NNP, RB, NN, etc. Figure 3 The dependency tags shown in the table are DET, nsubj, CONJ:ADN, obj:in, punct, obj, advmod, etc. The specific meanings of these tags can be found in the following table:
[0141]
[0142]
[0143] It should be noted that when performing dependency analysis on a text, it can be done with the help of an external dependency analysis tool, or it can be done by using the dependency analysis capability of the LLM model itself, which is not particularly limited in this specification. For example, in one example, in addition to using an external dependency analysis tool to complete the dependency analysis, the LLM model can also be fine-tuned based on a public data set to allow the LLM model to have the ability of dependency analysis, and then the LLM model that has been fine-tuned can be used to perform dependency analysis on the text.
[0144] In one embodiment shown, when extracting text events with dependencies contained in the above-mentioned text from the above-mentioned dependency tree, specifically only the text events contained in the above-mentioned text that have a direct or indirect dependency relationship with the specified text events can be extracted from the above-mentioned dependency tree, while other text events that do not have a direct or indirect dependency relationship with the specified text events are ignored. Then, based on the extracted text events that have a direct or indirect dependency relationship with the specified text events, a prompt text for describing the dependency relationship between the text events contained in the text is constructed as a guidance prompt corresponding to the above-mentioned text.
[0145] For example, see Figure 4 , Figure 4 A schematic diagram of a data sample shown in this specification.
[0146] like Figure 4 As shown, the data sample may include a text portion, an instruction portion, a guide prompt portion, and a label portion.
[0147] Among them, Figure 4 As shown, the text part can still be as follows:
[0148] “Last year more than 3,0000civilians were Killed and another 4,500 injured in Afghanistan,with roughly a5% increase from 2010”
[0149] The text contains three types of text events: killed event, injured event and increase event.
[0150] like Figure 4 As shown, the instruction part can be:
[0151] “List the parent event of "Sickened"
[0152] The meaning of this instruction is to extract the parent event of the killed event from the above text.
[0153] like Figure 4 As shown, the guide prompt part can be:
[0154] "Killed is the conj:and of injures ;
[0155] Killed is the object of increase ”
[0156] like Figure 4 As shown, the label part can be:
[0157] increase
[0158] against Figure 4 The dependency tree obtained by performing dependency analysis on the text part of the data sample shown can be seen in Figure 2 Based on Figure 2 It is not difficult to see from the dependency relationships shown that the dependency relationships contained in the above text part are very complex. Figure 4 The guide prompt part shown actually only retains Figure 2 In the dependency tree shown, there are two dependency branches that have direct dependencies on the Killed event.
[0159] Of course, in actual applications, in order to avoid missing some parent events of the Killed event, in actual applications, in addition to Figure 4In addition to the illustrated example where the guidance prompt portion only includes dependencies having a direct dependency relationship with the Killed event, the guidance prompt portion may also include dependencies having an indirect dependency relationship with the Killed event.
[0160] In this way, it can be ensured that the structured knowledge encapsulated in the above-mentioned guidance prompt part only includes text events that have a direct or indirect dependency relationship with the specified text events, and does not include other text events that do not have a direct or indirect dependency relationship with the specified text events. This allows the LLM model to avoid negative impacts on the reasoning results of the LLM model due to text events that do not have a direct or indirect dependency relationship with the specified text events when performing logical reasoning based on the above-mentioned guidance prompt part.
[0161] Step 104: inputting the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text;
[0162] In the specification, the data samples in the above-mentioned data sample set can be input into the LLM model as training samples, and the LLM model performs logical reasoning on the text contained in the data sample according to the instructions contained in the data sample and based on the guidance prompts contained in the data sample, so as to select at least one target text event from the text events contained in the text that has a text event relationship of the target type specified from the above-mentioned multiple types of text event relationships with the specified text event.
[0163] Among them, it should be noted that the specific process of the LLM model performing logical reasoning on the text contained in the data sample based on the guidance prompts contained in the data sample will not be described in detail in this specification; for example, in actual applications, the LLM model can perform logical reasoning on the text contained in the data sample based on the guidance prompts contained in the data sample, predict the probability of each text event contained in the text being a text event of the above-mentioned target type with the above-mentioned specified text event, and then select at least one text event with the above-mentioned highest probability from the various text events contained in the text as the final model output.
[0164] In an illustrated embodiment, the structured knowledge encapsulated in the guidance prompt may pre-include a set of candidate text events corresponding to the at least one target text event.
[0165] In this case, in the process of performing logical reasoning on the text contained in the data sample based on the guidance prompts contained in the data sample, LLM can specifically select the above-mentioned at least one target text event from the set of candidate text events contained in the structured knowledge encapsulated in the guidance prompts.
[0166] In this way, the amount of computation required by LLM when performing logical reasoning can be reduced.
[0167] The candidate text events in the above-mentioned candidate text event set may specifically be candidate text events that are pre-selected from text events contained in the text by pre-processing the text.
[0168] For example, in one example, as mentioned above, the structured knowledge encapsulated in the above guidance prompt may specifically include all text events with dependencies in the above text, or may only include text events that have direct or indirect dependencies with the above specified text events. In this case, these text events included in the structured knowledge encapsulated in the above guidance prompt may be used as a set of candidate text events corresponding to the above at least one target text event. In the process of performing logical reasoning on the text contained in the data sample based on the guidance prompt contained in the data sample, LLM may specifically select the above at least one target text event from these text events included in the structured knowledge encapsulated in the above guidance prompt.
[0169] It should be noted that, in addition to being included in the above-mentioned guidance prompts as structured knowledge, the above-mentioned candidate text event set can also be used as a part of the data sample independent of the above-mentioned guidance prompts in actual applications and be separately encapsulated in the data sample, which is not specifically limited in this specification.
[0170] In one embodiment shown, in order to ensure that the LLM model can accurately identify text events contained in data samples, before the data samples in the data sample set are input into the LLM model as training samples, these data samples can also be preprocessed, and text event recognition can be performed on the data samples to determine the word elements corresponding to the text events contained in the data samples; then, preset identifiers can be added to the positions corresponding to these word elements contained in the data samples; wherein the preset identifier is specifically used to uniquely identify the text events contained in the data sample.
[0171] For example, still taking the following text as an example:
[0172] “Last year more than 3,0000civilians were Killed and another 4,500 injured in Afghanistan,with roughly a5% increase from 2010”
[0173] The text contains three types of text events, namely killed events, injured events and increase events. In order to facilitate the LLM model to identify these events, a unique identifier can be set as a preset identifier for each of the above-mentioned events, and the unique identifier can be added to the position of the word units corresponding to these events contained in the above text.
[0174] The text with the above preset identifiers added is as follows:
[0175] “Last year more than 3,0000civilians were� Killed and another 4,500� injured in Afghanistan,with roughly a 5%� increase from 2010”
[0176] Among them, � is a preset identifier for uniquely identifying a killed event, � is a preset identifier for uniquely identifying an injured event, and � is a preset identifier for uniquely identifying an increase event. It should be emphasized that the above listed identifiers are only illustrative and can be flexibly set in actual applications.
[0177] In one embodiment shown, the at least one annotated text event annotated in the data sample in the data sample set can be specifically represented by a type ID corresponding to the type of text event relationship actually existing between the at least one annotated text event and the specified text event.
[0178] The above-mentioned type ID may specifically be a type ID obtained by performing vocabulary table mapping on the above-mentioned type based on a preset vocabulary table.
[0179] For example, in one example, the above-mentioned type ID can specifically be an ID in numerical form. By mapping the above-mentioned at least one labeled text event into a numerical type ID, these labeled information used as labels can be converted into a form that is easier for the LLM model to understand and process, so that the model can process non-numerical category outputs.
[0180] In this way, the text events annotated for the data samples in the data sample set can be converted into a form that is easier for the LLM model to understand and process; for example, converted into a numerical ID that is easier for the LLM model to understand and process.
[0181] In one embodiment shown, the data samples in the above data sample set usually include a large number of positive samples and negative samples.
[0182] Among them, a positive sample refers to a data sample in which the text contains a text event that has an event relationship with the above-specified text event of the above-specified target type; conversely, a negative sample refers to a data sample in which the text contains no text event that has an event relationship with the above-specified text event of the above-specified target type.
[0183] However, since in this specification, the event relationship that the LLM model needs to extract is only a specific event relationship type specified from multiple event relationship types to be extracted, in actual applications, the number of texts in the data sample that have this specific event relationship type with the above-specified text event may be very small, which may cause an imbalance in the number of positive samples and negative samples in the above-mentioned data sample set.
[0184] For example, in practical applications, the number of positive samples contained in the above data sample set may be very small, but there will be a large number of negative samples. Too few positive samples may affect the training effect of fine-tuning the LLM model. On the contrary, too many negative samples may also affect the training efficiency of the LLM model.
[0185] In one embodiment shown, before the data samples in the data sample set are input into the LLM model as training samples, the data sample set can also be preprocessed, the negative samples in the data sample set can be downsampled, and at least part of the negative samples in the data sample set can be deleted to achieve the purpose of controlling the number of negative samples in the data sample set.
[0186] For example, in one example, a random deletion method can be adopted to randomly delete some negative samples included in the data sample set. The number of deleted negative samples can be set based on demand and is not particularly limited in this specification.
[0187] In another example, in addition to random deletion, invalid negative samples in the negative samples included in the data sample set can also be deleted in a targeted manner. Invalid negative samples can specifically refer to data samples in which no text event relationship of any form exists in the included text. In this case, when preprocessing the data sample set, invalid negative samples in the negative samples included in the data sample set can be identified, and these invalid negative samples determined from the data sample set can be deleted.
[0188] It should be noted that the identification method used when identifying invalid negative samples in the negative samples included in the data sample set is not particularly limited in this specification.
[0189] For example, in one example, since the pre-trained LLM model itself has certain reasoning capabilities, a zero-shot learning method can be used to identify invalid negative samples in the negative samples contained in the data sample set by using the reasoning capabilities of the pre-trained LLM model itself before fine-tuning the LLM model. For example, by constructing prompt words, the data samples in the data sample set can be input into the pre-trained LLM model, and the LLM model can perform logical reasoning on the data samples to identify invalid negative samples contained in the data sample set.
[0190] In one embodiment shown, before inputting the data samples in the data sample set as training samples into the LLM model, in addition to preprocessing the data sample set and deleting at least part of the negative samples in the data sample set, in practical applications, the positive samples contained in the data sample set can also be expanded to generate some new positive samples to expand the number of positive samples contained in the data sample set.
[0191] For example, in one example, new positive samples may be generated by replacing some words in the text of the positive samples in the data sample set with words with the same or similar semantics, so as to expand the number of positive samples in the data sample set.
[0192] In this case, text event recognition can be performed on the text contained in the positive sample in the data sample set to determine the word-grams contained in the text that correspond to the text event, and the remaining word-grams other than the word-grams corresponding to the text event contained in the above text can be replaced with word-grams with the same or similar semantics as these remaining word-grams to generate an extended positive sample corresponding to the positive sample; then, the generated extended positive sample can be added to the data sample set to expand the number of positive samples in the data sample set.
[0193] Of course, in actual applications, when replacing the remaining words other than the words corresponding to the text events contained in the above text with words that have the same or similar semantics as these remaining words, you can only select some keyword words from these remaining words, and then only replace these keyword words with words that have the same or similar semantics.
[0194] The specific method of selecting the key word from the remaining word elements can be flexibly selected and is not particularly limited in this specification.
[0195] For example, take the following text:
[0196] “Last year more than 3,0000civilians were Killed and another4, 500injured in Afghanistan,with roughly a5% increase from 2010”
[0197] The above text contains three text events, namely killed event, injured event and increase event. Killed, injured and increase are word-grams corresponding to the text events, and the word-grams other than these three word-grams in the text are remaining word-grams.
[0198] When expanding the above text, qualifier terms and cardinality terms such as 3,0000civilians and another 4,500 among the remaining terms can be selected as keyword terms, and then the term 3,0000civilians can be replaced with the semantically similar term ten soldiers, and the term another 4,500 can be replaced with the semantically similar term twenty.
[0199] The above text after token replacement is:
[0200] “Last year more than ten soldiers were Killed and twenty injured inAfghanistan,with roughly a 5% increase from 2010”
[0201] In another example, a mask-then-fill strategy may be used to generate new positive samples to expand the number of positive samples in the data sample set.
[0202] In this case, text event recognition can also be performed on the text contained in the positive sample in the data sample set to determine the word-grams contained in the text corresponding to the text event, and the remaining word-grams other than the word-grams corresponding to the text event contained in the text are replaced with mask identifiers. Of course, in actual applications, when the remaining word-grams other than the word-grams corresponding to the text event contained in the above text are replaced with mask identifiers, only some key word-grams can be selected from these remaining word-grams, and then only these key word-grams are replaced with mask identifiers.
[0203] After the replacement is completed, the positive sample can be further input into the pre-trained LLM model so that the LLM model can perform logical reasoning on the positive sample and predict the word element corresponding to the above mask identifier. Then, the above mask identifier can be replaced with the word element corresponding to the above mask identifier predicted by the LLM model, that is, the predicted word element is filled into the position of the mask identifier to generate an extended positive sample corresponding to the positive sample; finally, the generated extended positive sample can also be added to the data sample set to expand the number of positive samples in the data sample set.
[0204] For example, still taking the following text as an example:
[0205] “Last year more than 3,0000civilians were Killed and another 4, 500injured in Afghanistan,with roughly a5% increase from 2010”
[0206] When expanding the above text, we can select the remaining tokens such as 3,0000civilians and another 4,500 as key tokens, and then replace the tokens 3,0000civilians and another 4,500 with mask identifiers:
[0207] “Last year more than [Mask] were Killed and [Mask]injured inAfghanistan,with roughly a 5% increase from 2010”
[0208] Finally, the text after word-unit replacement can be input into a pre-trained LLM model, and the LLM model uses its own reasoning ability to predict the word-unit corresponding to the mask identifier.
[0209] Assuming that the predicted word corresponding to the mask identifier of the word 3,0000civilians is tensoldiers, and the word corresponding to the mask identifier of the word another 4,500 is twenty, the mask identifiers contained in the above text can be replaced one by one with the word corresponding to the mask identifier predicted by the LLM model, so as to fill the predicted word into the position of the mask identifier. At this time, the text after the replacement is completed is:
[0210] “Last year more than ten soldiers were Killed and twenty injured inAfghanistan,with roughly a 5% increase from 2010”
[0211] Step 106: obtaining the at least one target text event output by the LLM model, and calculating the error loss between the at least one target text event and the at least one annotated text event; and, rearranging and combining the at least one target text event to obtain a plurality of text event sets, and bilaterally matching the plurality of text event sets with annotated text event sets formed by the at least one annotated text event, respectively, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculating the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set;
[0212] See also Figure 5 , Figure 5 This is a framework diagram for fine-tuning the LLM model shown in this specification.
[0213] like Figure 5As shown, after a data sample is input into the LLM model, the LLM model performs logical reasoning on the text contained in the data sample based on the guidance prompt part, and selects at least one target text event from the text events contained in the text that has a text event relationship of a target type specified from the above-mentioned multiple types of text event relationships with a specified text event, the above-mentioned at least one target text event selected by the LLM model can be obtained, and a model loss can be constructed based on the above-mentioned at least one target text event obtained.
[0214] Among them, Figure 5 As shown, the model loss constructed for the LLM model may specifically include the calculated error loss between the at least one target text event and the at least one annotated text event.
[0215] It should be noted that the specific representation method of the error loss between the at least one target text event and the at least one annotated text event is not particularly limited in this specification.
[0216] In one embodiment shown, a cross entropy loss may be used to characterize the error loss between the at least one target text event and the at least one annotated text event.
[0217] It needs to be explained that the cross entropy loss can usually be used to measure the difference between the probability distribution of the model output and the probability distribution of the true label, thereby evaluating the prediction accuracy of the model.
[0218] For example, in one example, the error loss can be represented by a cross entropy loss between a first probability distribution of at least one target text event and a second probability distribution corresponding to at least one annotated text event selected from text events contained in the text by performing logical reasoning on the text contained in the data sample based on the guidance prompt.
[0219] In this case, when calculating the error loss between the at least one target text event and the at least one annotated text event, it can be specifically calculated based on the above-mentioned guidance prompt to perform logical reasoning on the text contained in the data sample to select a first probability distribution corresponding to the at least one target text event from the text events contained in the text, and then use the cross-entropy loss function to further calculate the cross-entropy loss between the first probability distribution and the second probability distribution corresponding to the at least one annotated text event.
[0220] For example, the above cross entropy loss can be expressed as follows:
[0221]
[0222] In the above formula, L CE represents the cross entropy loss, that is, the error loss between the at least one target text event and the at least one annotated text event; N1 represents the character length of the guide prompt part contained in the data sample, that is, assuming that the guide prompt part is a string consisting of N1 characters; CE represents the cross entropy loss function; y i represents the i-th text event in the at least one target text event; i represents the i-th labeled text event in the at least one labeled text event mentioned above; since z i is a label item, so it is similar to z i The corresponding probability distribution can be expressed as z i itself to represent; x represents the above-mentioned guidance prompt part, P(y i |x) can represent the probability distribution composed of the conditional probability of each character in the above-mentioned guidance prompt part for the i-th text event in the above-mentioned at least one target text event; specifically, X can be represented as a string composed of N1 characters, such as X=[X 1 , X 2 …X N1 ], correspondingly, P(y i |x)=[P(y i |X 1 ), P(y i |X 2 )…P(y i |X N1 )], that is, P(y i |x) is composed of [X 1 , X 2 …X N1 ] for each character in y i A probability distribution consisting of the conditional probabilities of .
[0223] It should be noted that in the above cross entropy loss formula, the cumulative value of the cross entropy loss of each character in the guidance prompt part is used, and then the average value is obtained by dividing it by the length of the characters contained in the guidance prompt part to represent the above error loss. In practical applications, only the cumulative value of the cross entropy loss of each character in the guidance prompt part can be used to represent the above error loss.
[0224] For example, in another example, the above cross entropy loss can also be expressed as follows:
[0225]
[0226] The meanings of the parameters in the above formula remain unchanged and will not be repeated here.
[0227] It needs to be explained that the conditional probability of a certain character in the above-mentioned guidance prompt part for the i-th text event in the above-mentioned at least one target text event specifically refers to the probability of correctly predicting the occurrence of the i-th text event in the above-mentioned at least one target text event when the event of the character as a character in the guidance prompt part occurs.
[0228] Of course, in practical applications, when the above cross entropy loss is expressed using a formula, the conditional probability mentioned above can also be replaced by other forms of probability expression. For example, the conditional probability mentioned above can also be replaced by a posterior probability, or other forms of probability, which are not particularly limited in this specification.
[0229] It should be noted that, in addition to using cross entropy loss to characterize the error loss between the at least one target text event and the at least one annotated text event, other types of loss functions can also be used to characterize the error loss between the at least one target text event and the at least one annotated text event.
[0230] For example, in one example, the margin loss may be used to characterize the error loss between the at least one target text event and the at least one annotated text event, which will not be listed one by one in this specification.
[0231] Please continue to see Figure 5 The constructed model loss may include, in addition to the calculated error loss between the at least one target text event and the at least one annotated text event, a bilateral matching loss obtained by performing bipartite matching between the at least one target text event and the at least one annotated text event.
[0232] It should be explained that bilateral matching, also known as bipartite graph matching or bipartite graph matching, is an important concept in graph theory. It refers to finding a set of edges in a bipartite graph so that there are no common vertices between any two edges, and the number of edges is as large as possible. A bipartite graph means that the vertices can be divided into two disjoint sets, and the two endpoints of each edge belong to these two disjoint sets respectively.
[0233] In this specification, when calculating the bilateral matching loss obtained by performing bipartite matching on the at least one target text event and the at least one annotated text event, firstly, the text events in the at least one target text event may be rearranged and combined to obtain a plurality of text event sets;
[0234] For example, in one example, assuming that the at least one target text event includes three text events, namely text event A, text event B, and text event C, the text event set obtained by rearranging and combining these text events may include sets such as [A], [B], [C], [A, B], [A, C], [B, C], and [A, B, C].
[0235] Furthermore, the above-mentioned several text event sets can be bilaterally matched with the annotated text event set composed of the above-mentioned at least one annotated text event, so as to match the optimal text event set corresponding to the above-mentioned annotated text event set from the above-mentioned several text event sets. The above-mentioned optimal text event set is usually an event set whose text events have the highest matching degree with the text events contained in the above-mentioned annotated text event set.
[0236] For example, based on the bilateral matching algorithm, when two sets are bilaterally matched, the same elements contained in the two sets are usually connected by an edge. The goal of bilateral matching is to find the optimal set with the largest number of edges with a target set. In other words, the optimal set corresponding to the target set specifically refers to the set with the largest number of elements that are the same as the target set without considering the order of the elements in the set.
[0237] It should be noted that the bilateral matching algorithm used when bilaterally matching the above-mentioned several text event sets with the annotated text event set composed of the above-mentioned at least one annotated text event is not specifically limited in this specification; for example, in one example, the Hungarian Algorithm can be used.
[0238] After an optimal text event set corresponding to the annotated text event set is matched from the plurality of text event sets through bipartite graph matching, a bilateral matching loss may be calculated based on the optimal text event set.
[0239] In one embodiment shown, the probability value of the target text event included in the above optimal text event set and the target annotated text event included in the above annotated text event set being the same text event can be calculated; wherein the above target annotated text event can be an annotated text event with the same position order as the target text event in the above optimal text event set. For example, assuming that the above target text event is the i-th text event in the above optimal text event set, the above target annotated text event is the i-th annotated text event in the above annotated text event set.
[0240] Then, the bilateral matching loss can be constructed based on the probability value. For example, the bilateral matching loss can be constructed based on the probability value that the i-th text event included in the optimal text event set is the same as the i-th annotated text event included in the annotated text event set.
[0241] When constructing the bilateral matching loss, by introducing the above probability value, the event relationship extraction capability of the LLM model can be optimized, enabling the LLM model to select at least one annotated text event from the text events contained in the text whose position order is exactly the same as the standard text event in the above annotated text event set, thereby improving the accuracy of the LLM model in extracting event relationships.
[0242] It should be noted that when constructing the bilateral matching loss based on the above probability value, the specific form of the loss function used is not particularly limited in this specification.
[0243] For example, in one embodiment shown, a negative log-likelihood loss function may be used to construct a bilateral matching loss. In this case, the bilateral matching loss may be expressed as follows:
[0244]
[0245] Among them, L bpm represents the bilateral matching loss; M represents the number of the at least one target text event selected from the text events contained in the text; θ(i) represents the i-th target text event contained in the above optimal text event set; c i represents the i-th target annotated text event contained in the above annotated text event set; p θ(i) (c i ) represents the probability that the i-th target text event included in the above optimal text event set is the same text event as the i-th target annotated text event included in the above annotated text event set.
[0246] The idea behind the above formula is to introduce a negative log-likelihood function -logp θ(i) (c i ) to characterize the bilateral matching loss, and by adding a 1 as a subtrahend before the negative log-likelihood function, the loss value of the negative log-likelihood function can be ensured to be positive.
[0247] It should be emphasized that the idea of constructing the above-mentioned bilateral matching loss by introducing a negative log-likelihood function is only exemplary. In practical applications, other construction ideas can obviously be adopted to construct the bilateral matching loss using other forms of loss functions, which will not be listed one by one in this manual.
[0248] Step 108: Using the error loss and the bilateral matching loss as model losses, adjusting the model parameters included in the LLM model to complete supervised fine-tuning training for the LLM model.
[0249] In this specification, after constructing the above-mentioned error loss and the above-mentioned bilateral matching loss, the above-mentioned error loss and the above-mentioned bilateral matching loss can be used as model losses to adjust the model parameters contained in the LLM model to complete supervised fine-tuning training for the above-mentioned LLM model.
[0250] For example, in one example, after constructing the error loss and the bilateral matching loss, a weighted calculation can be performed on the error loss and the bilateral matching loss, and the loss after the weighted calculation is used as the final model loss, and the model parameters included in the LLM model are adjusted to complete the supervised fine-tuning training for the LLM model. Assume that the error loss is expressed as L CE Indicates that the above bilateral matching loss is expressed by L bpm Represented, the final model loss can be expressed as follows:
[0251] L=αL CE +βL bpm
[0252] Among them, L represents the final model loss; α and β represent L CE and L bpm It should be noted that the specific values of α and β can be flexibly set based on actual needs in practical applications and are not particularly limited in this specification.
[0253] The above-mentioned error loss and the above-mentioned bilateral matching loss are used as model losses, and the model parameters contained in the LLM model are adjusted to complete the specific process of supervised fine-tuning training for the above-mentioned LLM model, which will not be described in detail in this manual.
[0254] In the present specification, after completing the fine-tuning training for the above-mentioned LLM model, the task of extracting a text event relationship having a specified target type with a specified text event from a text using LLM can be converted into a MAQ task of selecting at least one target text event relationship having a specified target type with a specified text event from text events contained in the text using LLM. Subsequently, the LLM model can be used to select at least one target text event relationship having a specified target type with a specified text event from text events contained in the text in the manner of executing the MAQ task.
[0255] See also Figure 6 , Figure 6 The flowchart of a method for extracting text event relations shown in this specification includes the following execution process:
[0256] Step 602: Acquire a data sample for event relationship extraction; wherein the data sample includes text and instructions for instructing to extract at least one target text event from the text; the target text event is a text event having a target type of text event relationship with a specified text event; the target type of text event relationship is a text event relationship type specified from multiple types of text event relationships;
[0257] In the present specification, after completing the fine-tuning training for the above-mentioned LLM model, the task of extracting a text event relationship having a specified target type with a specified text event from a text using LLM can be converted into a MAQ task of selecting, from text events contained in the text using LLM, at least one target text event relationship having a text event relationship of a specified target type from multiple types of text event relationships to be extracted.
[0258] Among them, the multiple types of text event relations to be extracted can still include the co-reference relations, temporal relations, causal relations, parent-child relations, etc. mentioned above. The above-mentioned specified text event refers to a basic text event specified from various text events contained in the text that needs to extract the text event relations. The above-mentioned specified target type of text event relations refers to a certain text event relation that needs to be extracted from the above-mentioned multiple types of text event relations to be extracted.
[0259] When event relationship extraction is performed using the LLM model that has been fine-tuned and trained, a data sample for event relationship extraction can be obtained; wherein the data structure of the data sample is similar to the data structure of the data sample used as a training sample for fine-tuning the LLM model, and can still contain text, as well as instructions for extracting from the text at least one target text event that has a text event relationship of the above-specified target type with a specified text event.
[0260] It should be noted that, compared with the data format of the data sample used as the training sample when fine-tuning the LLM model, the data sample used for event relationship extraction may no longer need to include at least one annotated text event pre-annotated for the above text. The specific format of the above instruction is not specifically limited in this specification. In practical applications, different instruction formats can be used for different types of text event relationships to be extracted; for example, the specific format of the above instruction can be found in Figure 2 shown.
[0261] Step 604: inputting the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects the at least one target text event from the text events contained in the text, and completes event relationship extraction for the text;
[0262] The LLM model can be specifically an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose included text events have the highest matching degree with the text events included in the annotated text event set.
[0263] After obtaining the data sample for event relationship extraction, the data sample can be input into the above-mentioned LLM model that has been fine-tuned and trained. The LLM model performs logical reasoning on the text contained in the data sample according to the instructions contained in the data sample, and selects at least one target text event from the text events contained in the text in a manner of executing the MAQ task.
[0264] In one embodiment shown, in order to allow the LLM model to have sufficient reference information during the process of reasoning based on data samples, the above data samples may include not only text for instructing to extract from the text at least one target text event that has a text event relationship of a target type specified from the multiple types of text event relationships with a specified text event, but also, in actual applications, may include a guiding prompt for guiding the LLM model to select the above at least one target text event from the text events contained in the text.
[0265] In this case, when the LLM model performs logical reasoning on the text contained in the above data sample, it can specifically perform logical reasoning on the text based on the guidance prompt to select the above at least one target text event from the text events contained in the text.
[0266] In practical applications, by including the above-mentioned guidance prompts as part of the content included in the above-mentioned data samples, that is, as part of the content in the input of the LLM model, although sufficient reference information can be provided to the LLM model to assist the LLM model in logical reasoning, encapsulating too much content in the input of the LLM model will cause the text length of the above-mentioned data samples to increase sharply, which may weaken the accuracy of the reasoning results of the LLM model. For example, in the above-mentioned guidance prompt part, a large amount of structured knowledge may need to be encapsulated, and this structured knowledge may cause the text length of the above-mentioned data samples to increase sharply, which may weaken the accuracy of the reasoning results of the LLM model.
[0267] Based on this, in one embodiment shown, the above-mentioned guidance prompts can also be encapsulated in the output of the LLM model as part of the output of the LLM model; for example, the above-mentioned guidance prompts can be used as intermediate reasoning results of the LLM model's logical reasoning on the text.
[0268] In this case, the above-mentioned data sample may no longer contain the above-mentioned guiding prompts. When the LLM model performs logical reasoning on the text contained in the above-mentioned data sample, it can first perform preliminary logical reasoning on the text to generate a guiding prompt for guiding the LLM model to select the above-mentioned at least one target text event from the text events contained in the text.
[0269] It should be noted that, in the stage of fine-tuning the LLM model, the data sample used as the training sample still needs to include the above-mentioned guidance prompts, that is, in the stage of fine-tuning the LLM model, the above-mentioned guidance prompts still need to be encapsulated in the input of the LLM model. In this way, the LLM model can learn the data structure of the above-mentioned guidance prompts in the fine-tuning training stage, so that in the stage of extracting event relations using the LLM model that has completed fine-tuning training, when the data sample input to the LLM model for event relation extraction does not include the above-mentioned guidance prompts, the LLM model can temporarily generate the guidance prompts based on the knowledge learned in the fine-tuning stage during the process of logical reasoning for the data sample.
[0270] After the guiding prompt is generated, the LLM model can further perform logical reasoning on the text based on the guiding prompt to select the at least one target text event from the text events contained in the text.
[0271] By encapsulating the above-mentioned guidance prompts as the intermediate reasoning results of the LLM model's logical reasoning on the text in the output of the LLM model, the text length of the input data sample can be significantly reduced, which helps to improve the reasoning performance of the LLM model.
[0272] The specific structure and content of the above-mentioned guidance prompts will not be described in detail in this embodiment, and the details can be referred to the records in the previous embodiments.
[0273] For example, in one embodiment shown, the above-mentioned guidance prompt can specifically be a prompt text for describing the dependency relationship between text events contained in the text; in this case, when generating the guidance prompt, the LLM model can specifically perform dependency analysis on the text to obtain a dependency tree for describing the dependency relationship between text events contained in the text; then, the text events with dependency relationships contained in the text can be extracted from the dependency tree, and based on the extracted text events, a prompt text for describing the dependency relationship between text events contained in the text can be constructed as the guidance prompt corresponding to the text.
[0274] In one embodiment shown, when extracting text events with dependencies contained in the text from the dependency tree, it is also possible to only extract text events contained in the text that have a direct or indirect dependency relationship with the above-mentioned specified text events from the above dependency tree, and then construct a prompt text for describing the dependency relationship between the text events contained in the text based on the extracted text events, as the guidance prompt corresponding to the text.
[0275] In one embodiment shown, when the LLM model performs logical reasoning on the text based on the above-mentioned guidance prompt to select the above-mentioned at least one target text event from the text events contained in the text, specifically selects at least one target text event that has the above-mentioned target type of text event relationship with the above-mentioned specified text event from the text events contained in the above-mentioned guidance prompt.
[0276] For example, as mentioned above, the structured knowledge encapsulated in the above-mentioned guidance prompt may pre-include a set of candidate text events corresponding to the above-mentioned at least one target text event. In this case, the LLM model can specifically select at least one target text event that has the above-mentioned target type of text event relationship with the above-mentioned specified text event from the set of candidate text events contained in the above-mentioned guidance prompt.
[0277] Based on the above technical solution, on the one hand, by constructing a data sample set corresponding to various types of text event relations to be extracted, and fine-tuning the LLM model based on the data sample set, the LLM model can have the general extraction capability of extracting various types of text event relations from the text.
[0278] In this way, it is no longer necessary to train models for each type of text event relationship, which can significantly reduce the number of training samples required for fine-tuning the LLM model. For example, the number of training samples required can be reduced from O(n 2 ) is reduced to O(n).
[0279] On the other hand, by converting the task of extracting a text event relationship having a specified target type with a specified text event from a text using LLM into a question-answering task of selecting at least one target text event relationship having a text event relationship having a specified target type with a specified text event from text events contained in the text using LLM, the LLM model can be enabled to simultaneously extract multiple target text events having a text event relationship having a specified target type with a specified text event from a text.
[0280] In this way, the problem of missing some text events to be extracted due to the LLM model failing to cover all types of text event relations to be extracted can be avoided during the process of extracting text event relations, thereby improving the accuracy of extracting text event relations from text using the LLM model.
[0281] For example, in practical applications, the LLM model can be used to extract all text events that have a specified target type of text event relationship contained in the text without missing some text events that need to be extracted.
[0282] Thirdly, during the fine-tuning training of the LLM model, bilateral matching loss can also be introduced into the model loss.
[0283] Since bilateral matching is a matching algorithm that does not need to consider the arrangement order; therefore, by introducing bilateral matching loss into the model loss, the influence of the arrangement order of at least one target text event selected by the LLM model from the text events contained in the text on the extraction results of the LLM model can be reduced, thereby further improving the accuracy of the LLM model in extracting text event relations from the text.
[0284] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.
[0285] Figure 7 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 6 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0286] like Figure 8 As shown, Figure 8 is a block diagram of a text event relationship extraction device according to an exemplary embodiment of the present specification, which can be applied to Figure 7 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device 800 includes:
[0287] The first acquisition module 801 acquires a data sample set for fine-tuning the LLM model; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text and having a text event relation of the target type with the specified text event; the target text event is a text event having a text event relation of the target type with the specified text event; the text event relation of the target type is a text event relation type specified from the multiple types of text event relations;
[0288] A first input module 802 inputs the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text;
[0289] The calculation module 803 obtains the at least one target text event output by the LLM model, and calculates the error loss between the at least one target text event and the at least one annotated text event; and rearranges and combines the at least one target text event to obtain a plurality of text event sets, and performs bilateral matching on the plurality of text event sets with the annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculates the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set;
[0290] The training module 804 uses the error loss and the bilateral matching loss as model losses, and performs parameter adjustment on the model parameters included in the LLM model to complete supervised fine-tuning training for the LLM model.
[0291] like Fig. 9 As shown, Fig. 9 is a block diagram of another text event relationship extraction device according to an exemplary embodiment of the present specification, which can be applied to Figure 7 In the electronic device shown in the figure, the technical solution of this specification is implemented
[0292] The second acquisition module 901 acquires a data sample for event relationship extraction; wherein the data sample includes text and an instruction for instructing to extract at least one target text event from the text; the target text event is a text event having a text event relationship of a target type with a specified text event; the text event relationship of the target type is a text event relationship type specified from the multiple types of text event relationships;
[0293] The second input module 902 inputs the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects the at least one target text event from the text events contained in the text, and completes event relationship extraction for the text;
[0294] The LLM model is an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose text events contained therein have the highest matching degree with the text events contained in the annotated text event set.
[0295] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.
[0296] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.
[0297] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.
[0298] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0299] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C2051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0300] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, it is not excluded that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0301] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.
[0302] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0303] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0304] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0305] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0306] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0307] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0308] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0309] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0310] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0311] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.
[0312] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.
Claims
1. A method for extracting text event relations, the method comprising: A data sample set for fine-tuning an LLM model is obtained; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text and having a text event relation of a target type with a designated text event; the target text event is a text event having a text event relation of a target type with the designated text event; the text event relation of the target type is a text event relation type designated from the multiple types of text event relations; Inputting the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text; Acquire the at least one target text event output by the LLM model, and calculate the error loss between the at least one target text event and the at least one annotated text event; and rearrange and combine the at least one target text event to obtain a plurality of text event sets, and perform bilateral matching on the plurality of text event sets with annotated text event sets composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculate the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set; The error loss and the bilateral matching loss are used as model losses, and the model parameters included in the LLM model are adjusted to complete supervised fine-tuning training for the LLM model.
2. The method according to claim 1, wherein the guiding prompt is a prompt text for describing the dependency relationship between text events contained in the text; The method further comprises: Performing dependency analysis on the text to obtain a dependency tree for describing dependency relationships between text events contained in the text; The text events with dependency relationships contained in the text are extracted from the dependency tree, and prompt text for describing the dependency relationships between the text events contained in the text is constructed based on the extracted text events to serve as the guidance prompt corresponding to the text.
3. The method according to claim 2, wherein extracting text event pairs with dependency contained in the text from the dependency tree comprises: A text event contained in the text and having a direct or indirect dependency relationship with the designated text event is extracted from the dependency tree.
4. The method according to claim 1 or 3, wherein the logical reasoning is performed on the text based on the guiding prompt, and the at least one target text event is selected from the text events contained in the text, comprising: Based on the guidance prompt, logical reasoning is performed on the text, and at least one target text event having a text event relationship of the target type with the designated text event is selected from the text events included in the guidance prompt.
5. The method according to claim 1, before inputting the data samples in the data sample set as training samples into the LLM model, further comprising: Performing text event recognition on the data sample to determine the word-grams corresponding to the text event contained in the data sample; A preset identifier is added at a position corresponding to the word element contained in the data sample; wherein the preset identifier is used to uniquely identify the text event contained in the data sample.
6. The method as claimed in claim 4, wherein the error loss is represented by a cross entropy loss between a first probability distribution of at least one target text event and a second probability distribution corresponding to the at least one annotated text event, by performing logical reasoning on the text based on the guidance prompt to select the at least one target text event from the text events contained in the text.
7. The method according to claim 1, wherein the bilateral matching loss is a loss constructed based on the probability value that the target text event included in the optimal text event set is the same as the target annotated text event included in the annotated text event set; wherein, The target annotated text event is an annotated text event in the annotated text event set that has the same position order as the target text event in the optimal text event set.
8. The method of claim 1, wherein the bilateral matching loss is expressed by the following formula: in, L bpm represents the bilateral matching loss; M represents the number of the at least one target text event selected from the text events contained in the text; θ(i) represents the i-th target text event contained in the optimal text event set; c i represents the i-th target annotated text event contained in the annotated text event set; p θ(i) (c i ) represents the probability that the i-th target text event included in the optimal text event set is the same text event as the i-th target annotated text event included in the annotated text event set.
9. The method according to claim 8, wherein the at least one annotated text event contained in the data sample is represented by a type ID corresponding to the type of the text event relationship between the at least one annotated text event and the specified text event; wherein, The type ID is a type ID obtained by performing vocabulary mapping on the type based on a preset vocabulary.
10. The method according to claim 1, wherein the data samples in the data sample set include positive samples and negative samples; wherein: The positive sample refers to a data sample in which the text contains a text event having an event relationship of the specified target type with the specified text event; the negative sample refers to a data sample in which the text contains no text event having an event relationship of the specified target type with the specified text event; Before inputting the data samples in the data sample set as training samples into the LLM model, the method includes: Randomly delete some negative samples contained in the data sample set; or, Invalid negative samples in the negative samples included in the data sample set are identified, and the invalid negative samples identified from the data sample set are deleted; wherein the invalid negative samples are data samples in which no text event relationship of any form exists in the included text.
11. The method according to claim 10, wherein identifying invalid negative samples in the negative samples contained in the data sample set comprises: The data samples in the data sample set are input into the pre-trained LLM model so that the LLM model performs logical reasoning on the data samples to identify the invalid negative samples contained in the data sample set.
12. The method according to claim 10, before inputting the data samples in the data sample set as training samples into the LLM model, comprising: Performing text event recognition on the text contained in the positive sample in the data sample set to determine the word element contained in the text corresponding to the text event; Replacing the remaining word-grams contained in the text other than the word-grams corresponding to the text event with word-grams having the same semantics as or similar semantics to the remaining word-grams, so as to generate an extended positive sample corresponding to the positive sample; The extended positive samples are added to the data sample set to expand the number of positive samples in the data sample set.
13. The method according to claim 10, before inputting the data samples in the data sample set as training samples into the LLM model, comprises: Performing text event recognition on the text contained in the positive sample in the data sample set to determine the word element contained in the text corresponding to the text event; Replacing the remaining word-grams contained in the text except the word-grams corresponding to the text event with mask identifiers, and inputting the positive sample after the replacement into the pre-trained LLM model, so that the LLM model performs logical reasoning on the positive sample and predicts the word-grams corresponding to the mask identifier; Replacing the mask identifier with a word element corresponding to the mask identifier predicted by the LLM model to generate an extended positive sample corresponding to the positive sample; The extended positive samples are added to the data sample set to expand the number of positive samples in the data sample set.
14. The method according to claim 13, wherein the multiple types of event relationships include a combination of the following: Co-reference relationship, temporal relationship, causal relationship, and father-child relationship.
15. The method according to claim 1, wherein the text contained in the data sample is unstructured text.
16. The method according to claim 1, wherein the text contained in the data sample is a document.
17. A method for extracting text event relations, the method comprising: Acquire a data sample for event relationship extraction; wherein the data sample comprises text and an instruction for instructing to extract at least one target text event from the text; the target text event is a text event having a target type of text event relationship with a specified text event; the target type of text event relationship is a text event relationship type specified from multiple types of text event relationships; Inputting the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects at least one target text event from the text events contained in the text, and completes event relationship extraction for the text; The LLM model is an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose text events contained therein have the highest matching degree with the text events contained in the annotated text event set.
18. The method according to claim 17, wherein the data sample further comprises a guiding prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text; Performing logical reasoning on the text and selecting at least one target text event from text events contained in the text includes: Based on the guiding prompt, logical reasoning is performed on the text, and at least one target text event is selected from text events contained in the text.
19. The method of claim 17, wherein the data sample does not include a guiding prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text; Performing logical reasoning on the text and selecting at least one target text event from text events contained in the text includes: Logical reasoning is performed on the text to generate a guidance prompt for guiding the LLM model to select at least one target text event from the text events contained in the text, and logical reasoning is further performed on the text based on the guidance prompt to select at least one target text event from the text events contained in the text.
20. The method according to claim 19, wherein the guiding prompt is a prompt text for describing the dependency relationship between text events contained in the text; The generating of a guidance prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text includes: Performing dependency analysis on the text to obtain a dependency tree for describing dependency relationships between text events contained in the text; The text events with dependency relationships contained in the text are extracted from the dependency tree, and prompt text for describing the dependency relationships between the text events contained in the text is constructed based on the extracted text events to serve as the guidance prompt corresponding to the text.
21. The method of claim 20, wherein extracting text events with dependency contained in the text from the dependency tree comprises: A text event contained in the text and having a direct or indirect dependency relationship with the designated text event is extracted from the dependency tree.
22. The method according to claim 21, further performing logical reasoning on the text based on the guidance prompt to select the at least one target text event from the text events contained in the text, comprising: Based on the guidance prompt, logical reasoning is further performed on the text, and at least one target text event having a text event relationship of the target type with the designated text event is selected from the text events included in the guidance prompt.
23. A device for extracting text event relations, the device comprising: A first acquisition module acquires a data sample set for fine-tuning the LLM model; wherein the data samples in the data sample set correspond to multiple types of text event relations; the data sample comprises text, an instruction for instructing to extract at least one target text event from the text, a guide prompt for guiding the LLM model to select the at least one target text event from the text events contained in the text, and at least one annotated text event pre-annotated for the text and having a text event relation of a target type with a specified text event; the target text event is a text event having a text event relation of a target type with the specified text event; the text event relation of the target type is a text event relation type specified from the multiple types of text event relations; A first input module inputs the data samples in the data sample set as training samples into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text based on the guidance prompt, and selects the at least one target text event from the text events contained in the text; A calculation module, which obtains the at least one target text event output by the LLM model, and calculates the error loss between the at least one target text event and the at least one annotated text event; and rearranges and combines the at least one target text event to obtain a plurality of text event sets, and performs bilateral matching on the plurality of text event sets with annotated text event sets composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets, and calculates the bilateral matching loss based on the optimal text event set; wherein the optimal text event set is an event set whose text events have the highest matching degree with the text events contained in the annotated text event set; The training module uses the error loss and the bilateral matching loss as model losses, and performs parameter adjustment on model parameters included in the LLM model to complete supervised fine-tuning training for the LLM model.
24. A device for extracting text event relations, the device comprising: The second acquisition module acquires a data sample for event relationship extraction; wherein the data sample includes text and an instruction for instructing to extract at least one target text event from the text; the target text event is a text event having a text event relationship of a target type with a specified text event; the text event relationship of the target type is a text event relationship type specified from multiple types of text event relationships; A second input module inputs the data sample into the LLM model, so that the LLM model responds to the instruction, performs logical reasoning on the text, selects at least one target text event from the text events contained in the text, and completes event relationship extraction for the text; The LLM model is an LLM model obtained by supervised fine-tuning training based on a preset data sample set; the data samples in the data sample set correspond to multiple types of text event relations to be extracted; the text contained in the data samples in the data sample set is pre-annotated with at least one annotated text event that has a text event relationship of a target type specified from the preset multiple types of text event relations with the specified text event; the model loss used for supervised fine-tuning training of the LLM model includes error loss and bilateral matching loss; the error loss is used to represent the error between the at least one target text event selected by the LLM model from the text events contained in the text and the at least one annotated text event; the bilateral matching loss is used to represent the matching loss of performing bilateral matching on a plurality of text event sets obtained by rearranging and combining the at least one target text event, respectively, with an annotated text event set composed of the at least one annotated text event, so as to match an optimal text event set corresponding to the annotated text event set from the plurality of text event sets; the optimal text event set is an event set whose text events contained therein have the highest matching degree with the text events contained in the annotated text event set.
25. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 22 are implemented.
26. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 22.
27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 22 are implemented.
Citation Information
Patent Citations
Graph-based topic event detection method, apparatus and device
CN118114669A
Cited By
Event relation extraction method based on knowledge enhancement
CN121766415A
