Big language model representation-based discriminant entity relationship classification method

Abstract

The invention discloses a discriminant entity relationship classification method based on large language model representation, and belongs to the technical field of natural language processing and representation learning. The implementation method comprises the following steps: 1, acquiring an entity relationship data set; 2, obtaining text representation; 3, forming entity point-level representation; 4, obtaining solid surface level representation; 5, forming enhanced point set representation; 6, fusing the entity surface-level representation and the enhanced point-level entity representation to obtain entity pair multi-source fusion features; 7, constructing a prediction model by using the multi-source fusion features, and training and optimizing the prediction model by using the training objective function to obtain a trained prediction model; 8, inputting the test data into a trained prediction model, generating prediction distribution of the relation label set through a linear classifier and a softmax layer, and further obtaining an entity relation classification result of a discriminant; compared with the prior art, the method has the advantage that the classification accuracy in the context-aware entity relationship classification task is improved.

Description

Technical Field

[0001] This invention relates to a discriminative entity relation classification method based on large language model representation, belonging to the field of natural language processing and representation learning technology, and is applied to entity relation modeling and classification tasks in multiple scenarios. Background Technology

[0002] In recent years, with the development of internet and big data technologies, massive amounts of text data and information are produced every day. The vast majority of this data exists in the form of unstructured text, such as news reports, academic papers, social media content, and corporate documents. How to efficiently and accurately extract valuable structured knowledge from massive amounts of text has become a core challenge in the field of artificial intelligence, and entity relationship classification is an important component of information extraction.

[0003] Span-based entity relationships can represent information about different entities in entity relationship classification tasks, but this method cannot make accurate judgments based on contextual information, leading to poor entity relationship classification performance in some complex tasks. Large language models have powerful modeling capabilities for long contexts, and the knowledge information contained within them can further enhance the capabilities of downstream tasks. However, current methods are mainly generative, which is computationally inefficient and susceptible to the illusion problem, generating incorrect answers. Using a discriminative approach can avoid the shortcomings of generative methods while fully utilizing the knowledge information contained in large language models.

[0004] Therefore, in context-aware entity relationship classification tasks, how to accurately model the relationships between entity pairs in the context has become an urgent problem to be solved. Summary of the Invention

[0005] To address the technical challenge of improving classification accuracy while maintaining computational efficiency in entity relationship classification tasks within context-aware scenarios, this invention proposes a discriminative entity relationship classification method based on large language model representation. This method accurately models complex relationships in the form of a discriminative task by using entity information representation, contextual representation modeling, and cue-based learning representation enhancement.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a discriminative entity relation classification method based on large language model representation, comprising the following steps:

[0008] Step 1: Use an open-source dataset, add spans using an LLM tokenizer, and decode to obtain the entity relationship dataset;

[0009] Step 1.1: Transform the open-source dataset to extract entity, context, and relationship labels;

[0010] Step 1.2: Add entity span to entities based on the large language model word segmenter;

[0011] Step 1.3: After decoding the entity span, restore the entity, filter the data, and obtain the entity relationship dataset;

[0012] Step 2: Encode the text using a large language model to obtain a text representation;

[0013] Step 2.1: Construct a relation classification task instruction prompt using the entity relation dataset and append it to the end of the original text.

[0014] Step 2.2: Use the forward propagation method of the pre-trained language model to encode the information in the entity relation dataset and obtain text representations;

[0015] Step 3: Fuse the boundary features formed by the multilayer perceptron projection with the type vector to form a point-level representation of the entity;

[0016] Step 3.1: Project the initial and final position vectors of the target entity through a multilayer perceptron to obtain boundary features;

[0017] Step 3.1.1: In the encoded sequence, determine the vector representations of the start and end positions corresponding to the target entity;

[0018] Step 3.1.2: Project the vectors at the start and end positions through independent multilayer perceptrons to obtain boundary features;

[0019] Step 3.2: Map the discrete type encoding of the entity to a continuous vector and fuse it with the boundary features.

[0020] Step 3.3: Add the starting feature, ending feature and type vector together, and use the entity point-level representation obtained by formula (1);

[0021] e Point =MLP start (h start )+MLP end (h end )+e type (1)

[0022] Among them, h start h end These represent the start and end position vectors of the entity span, respectively; MLP start MLP end Represents an independent multilayer perceptron projection function; e type This represents the embedding vector obtained by encoding the entity type;

[0023] Step 4: Extract global semantic representation. Based on the causal attention mechanism of the pre-trained language model, obtain the vector of the last token at the end of the sequence as the entity face-level representation.

[0024] e Plain =h eos (2)

[0025] Step 5: Superimpose the entity line-level representation obtained by concatenating the attention scores with the entity point-level representation in the same latent space to form an enhanced point set representation;

[0026] Step 5.1: Calculate the attention score. Based on the causal attention mechanism, calculate the weighted correlation between the query matrix Q and the key matrix K to obtain the attention score matrix A as shown in equation (3).

[0027]

[0028] Where M is a causal mask, used to prevent the current position from focusing on future positions;

[0029] Step 5.2: Extract local and global relevance. Based on the attention score, calculate the average relevance between the [EOS] vector and the entity boundary and all tokens inside the entity as shown in Equation (4) as the line-level representation of the entity;

[0030]

[0031] Among them, A start and A end Let A represent the attention scores for the start and end positions of [EOS] and the span, respectively. i This represents the attention score between [EOS] and the i-th token within the span, where N is the number of tokens within the span. Concat represents concatenating the features of the input.

[0032] Step 5.3: Represent e at the line level Line Projected onto the same latent space as the point-level representation and added to the point-level representation, we obtain the enhanced point set entity representation as shown in Equation (5);

[0033] e Entity =e Point +MLP A (e Line (5)

[0034] Among them, MLP A This represents a linear projection network used to amplify the contribution of the linear representation.

[0035] Step 6: Fuse the entity surface-level representation and the enhanced point-level entity representation to obtain the entity pair multi-source fusion feature e as shown in Equation (6).Fused ;

[0036] e Fused =Concat(e Plane ,e Sub ,e Obj (6)

[0037] Among them, e Sub ,e Obj For the enhanced point-level entity representation e Entity The assigned subject and object entities are represented; e Plane It is a solid-level representation, which is spliced ​​together to form a fused feature with a dimension of 3*d.

[0038] Step 7: Construct a prediction model using multi-source fusion features, and train and optimize the prediction model using the training objective function shown in Equation (7) to obtain the trained prediction model;

[0039]

[0040] Where N represents the total number of samples, r i Let represent the true relation label of the i-th sample, and p represent the predicted probability based on the fusion representation.

[0041] Step 8: Input the test data into the trained prediction model, which generates a predicted distribution of the relation label set through a linear classifier and a softmax layer, thereby obtaining the discriminative entity relation classification result;

[0042] Compared with existing technologies, it has the following beneficial effects:

[0043] 1. This invention uses a large language model to encode and model the input text and instruction information, instead of using a small model. This method can fully utilize the knowledge information and contextual understanding capabilities of the current large model pre-training, and obtains richer text representation information compared to small model encoding, resulting in more accurate downstream relationship modeling.

[0044] 2. This invention uses an instruction enhancement method. After inputting natural language instructions related to the relation classification task as prompts into the original context, it utilizes the unidirectional convergence characteristic of large language models. The final token will converge the preceding input information, and the resulting representation will have a better relation classification perception ability compared to simply encoding the context.

[0045] 3. This invention uses a discriminative approach to model entity relationships, rather than the generative approach typically employed in large language models. Compared to generative methods, this method avoids the illusion problem and reduces the risk of classification errors. Furthermore, the discriminative approach, using a smaller language model, achieves better relationship classification results compared to the large-parameter language models used in generative methods, demonstrating the higher computational efficiency of this method.

[0046] 4. This method abstracts the information required for relation classification tasks into point information, surface information, and line information from the perspectives of geometry and information theory, and models them using a fusion method. Compared with commonly used methods that only use point information for modeling, this method provides richer information input and better interpretability. It provides comprehensive input for subsequent relation classification tasks, significantly improving the performance of relation classification. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the present invention; Detailed Implementation

[0048] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0049] Example

[0050] like Figure 1 As shown in the figure, the specific implementation steps of the discriminative entity relation classification method based on large language model representation in this embodiment are as follows:

[0051] Step 1: Use an open-source dataset, add spans using an LLM tokenizer, and decode to obtain the entity relationship dataset;

[0052] Step 1.1: Transform the open-source dataset to extract entity, context, and relationship labels;

[0053] Step 1.2: Add entity span to entities based on the large language model word segmenter;

[0054] Step 1.3: After decoding the entity span, restore the entity, filter the data, and obtain the entity relationship dataset;

[0055] In this embodiment, the present invention is based on a high-quality labeled dataset. The span information corresponding to the entity is obtained through data format conversion and word segmentation encoding. Then, the position corresponding to the span is decoded, the span is verified, and the final dataset is obtained after filtering.

[0056] Step 2: Encode the text using a large language model to obtain a text representation;

[0057] Step 2.1: Construct a relation classification task instruction prompt using the entity relation dataset and append it to the end of the original text.

[0058] Step 2.2: Use the forward propagation method of the pre-trained language model to encode the information in the entity relation dataset and obtain text representations;

[0059] In this embodiment, the present invention concatenates task-related natural language instructions to the end of the original input information. For example, after the original input "Judge entity A accuses bank entity B", the instruction "Please determine the relationship between entity A and entity B" is concatenated to form new input information. Then, the new input information is encoded by a large language model such as the "Thousand Questions" series of open-source models to obtain an encoded sequence.

[0060] Step 3: Fuse the boundary features formed by the multilayer perceptron projection with the type vector to form a point-level representation of the entity;

[0061] Step 3.1: Project the initial and final position vectors of the target entity through a multilayer perceptron to obtain boundary features;

[0062] Step 3.1.1: In the encoded sequence, determine the vector representations of the start and end positions corresponding to the target entity;

[0063] Step 3.1.2: Project the vectors at the start and end positions through independent multilayer perceptrons to obtain boundary features;

[0064] Step 3.2: Map the discrete type encoding of the entity to a continuous vector and fuse it with the boundary features.

[0065] Step 3.3: Add the starting feature, ending feature and type vector together, and use the entity point-level representation obtained by formula (1);

[0066] e Point =MLP start (h start )+MLP end (h end )+e type (1)

[0067] Among them, h start h end These represent the start and end position vectors of the entity span, respectively; MLP start MLP end Represents an independent multilayer perceptron projection function; e type This represents the embedding vector obtained by encoding the entity type;

[0068] In this embodiment, for entities A and B, the present invention locates the compressed representations of the start and end positions based on the corresponding spans, and fuses these compressed representations using a multilayer perceptron. Then, the entity categories are continuously transformed and added to the transformed compressed representations for fusion. After fusion, both entities A and B will obtain point-granular entity representation information.

[0069] Step 4: Extract global semantic representation. Based on the causal attention mechanism of the pre-trained language model, obtain the vector of the last token at the end of the sequence as the entity face-level representation.

[0070] e Plain =h eos (2)

[0071] In this embodiment, the present invention obtains the actual last position based on the attention mask, and obtains the information of the corresponding position in the compressed representation based on the position index as the global online information, i.e., surface information.

[0072] Step 5: Superimpose the entity line-level representation obtained by concatenating the attention scores with the entity point-level representation in the same latent space to form an enhanced point set representation;

[0073] Step 5.1: Calculate the attention score. Based on the causal attention mechanism, calculate the weighted correlation between the query matrix Q and the key matrix K to obtain the attention score matrix A as shown in equation (3).

[0074]

[0075] Where M is a causal mask, used to prevent the current position from focusing on future positions;

[0076] Step 5.2: Extract local and global relevance. Based on the attention score, calculate the average relevance between the [EOS] vector and the entity boundary and all tokens inside the entity as shown in Equation (4) as the line-level representation of the entity;

[0077]

[0078] Among them, A start and A end Let A represent the attention scores for the start and end positions of [EOS] and the span, respectively. i This represents the attention score between [EOS] and the i-th token within the span, where N is the number of tokens within the span. Concat represents concatenating the features of the input.

[0079] Step 5.3: Represent e at the line level Line Projected onto the same latent space as the point-level representation and added to the point-level representation, we obtain the enhanced point set entity representation as shown in Equation (5);

[0080] e Entity =e Point +MLP A (e Line (5)

[0081] Among them, MLP A This represents a linear projection network used to amplify the contribution of the linear representation.

[0082] In this embodiment, the present invention first obtains the attention weights between the last token and the left and right boundaries of each entity, as well as the attention weights between each token within the span, based on the causal attention score matrix generated by the forward encoding process. Next, these attention weights are concatenated to serve as the association information between the entity and its context, i.e., the line information. Then, the concatenated attention weight matrix is ​​transformed using a multilayer perceptron to amplify the role of the line information. Finally, the transformed line information is added to the point information to obtain the final entity representation.

[0083] Step 6: Fuse the entity surface-level representation and the enhanced point-level entity representation to obtain the entity pair multi-source fusion feature e as shown in Equation (6). Fused ;

[0084] e Fused =Concat(e Plane ,e Sub ,e Obj (6)

[0085] Among them, e Sub ,e Obj For the enhanced point-level entity representation e Entity The assigned subject and object entities are represented; e Plane It is a solid-level representation, which is spliced ​​together to form a fused feature with a dimension of 3*d.

[0086] Step 7: Construct a prediction model using multi-source fusion features, and train and optimize the prediction model using the training objective function shown in Equation (7) to obtain the trained prediction model;

[0087]

[0088] Where N represents the total number of samples, r i Let represent the true relation label of the i-th sample, and p represent the predicted probability based on the fusion representation.

[0089] Step 8: Input the test data into the trained prediction model, which generates a predicted distribution of the relation label set through a linear classifier and a softmax layer, thereby obtaining the discriminative entity relation classification result;

[0090] In this embodiment, the present invention first obtains the enhanced entity representation information of entity A and entity B based on formula (5). Then, the surface information representing context information is concatenated with the entity information as the input information for fusion. Then, the fused information is input into the relation classifier, and the relation category with the highest probability is output as the category predicted by the model. For example, in the TACRED dataset, there are 42 relations. The model will predict the probability of each category and select the category with the highest probability as the final category. Finally, the model is trained using cross-entropy as the optimization objective in the above manner. The trained model is used for downstream relation classification tasks. For example, among the 42 predicted relations between the input "Judge entity A accuses bank entity B", the highest probability is 0.87, and the corresponding category is "accusation". The model outputs the prediction result as "accusation".

[0091] To further illustrate the superiority of this invention, the following experimental results are presented in Table 1. As can be seen from the metrics of TACRED, TACREV, and RE-TACRED, the 0.6B model of this invention achieves scores of 88.9 and 92.8 on TACRED and TACREV, respectively, significantly outperforming other methods. The RE-TACRED score is comparable to other methods. With the increase of model parameters, the performance on each dataset continues to improve. Our 4B model achieves the best results, reaching scores of 89.9, 94.0, and 92.9 on the three datasets, respectively, significantly outperforming previous methods. In particular, the 0.4B model surpasses the 7B models of other methods, demonstrating the effectiveness of this invention. It fully utilizes the knowledge information and contextual understanding capabilities of the language model while possessing higher computational efficiency.

[0092] Table 1 Comparison of Model Performance

[0093]

[0094] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A discriminative entity relation classification method based on large language model representation, characterized in that: Includes the following steps, Step 1: Use an open-source dataset, add spans using an LLM tokenizer, and decode to obtain the entity relationship dataset; Step 2: Encode the text using a large language model to obtain a text representation; Step 3: Fuse the boundary features formed by the multilayer perceptron projection with the type vector to form a point-level representation of the entity; Step 4: Extract global semantic representation. Based on the causal attention mechanism of the pre-trained language model, obtain the vector of the last token at the end of the sequence as the entity face-level representation. e Plain =h eos (2) Step 5: Superimpose the entity line-level representation obtained by concatenating the attention scores with the entity point-level representation in the same latent space to form an enhanced point set representation; Step 5.1: Calculate the attention score. Based on the causal attention mechanism, calculate the weighted correlation between the query matrix Q and the key matrix K to obtain the attention score matrix A as shown in equation (3). Where M is a causal mask, used to prevent the current position from focusing on future positions; Step 5.2: Extract local and global relevance. Based on the attention score, calculate the average relevance between the [EOS] vector and the entity boundary and all tokens inside the entity as shown in Equation (4) as the line-level representation of the entity; Among them, A start and A end Let A represent the attention scores for the start and end positions of [EOS] and the span, respectively. i Represents the attention score between [EOS] and the i-th token within the span, where N is the number of tokens within the span; Concat represents concatenating the input features; Step 5.3: Represent e at the line level Line Projected onto the same latent space as the point-level representation and added to the point-level representation, we obtain the enhanced point set entity representation as shown in Equation (5); e Entity =e Point +MLP A (e Line ) (5) Among them, MLP A This represents a linear projection network used to amplify the contribution of the line-level representation; Step 6: Fuse the entity surface-level representation and the enhanced point-level entity representation to obtain the entity pair multi-source fusion feature e as shown in Equation (6). Fused ; And Fused =Concat(e Plane ,And Sub ,And Obj ) (6) Among them, e Sub ,e Obj For the enhanced point-level entity representation e Entity The assigned subject and object entities are represented; e Plane It is a solid-level representation, which is then stitched together to form a fused feature with a dimension of 3*d; Step 7: Construct a prediction model using multi-source fusion features, and train and optimize the prediction model using the training objective function shown in Equation (7) to obtain the trained prediction model; Where N represents the total number of samples, r i Let represent the true relation label of the i-th sample, and p represent the predicted probability based on the fusion representation; Step 8: Input the test data into the trained prediction model, which generates a predicted distribution of the relation label set through a linear classifier and a softmax layer, thereby obtaining the discriminative entity relation classification result.

2. The discriminative entity relation classification method based on a large language model representation as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Transform the open-source dataset to extract entity, context, and relationship labels; Step 1.2: Add entity span to entities based on the large language model word segmenter; Step 1.3: After decoding the entity span, restore the entity and filter the data to obtain the entity relationship dataset.

3. The discriminative entity relation classification method based on a large language model representation as described in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Construct a relation classification task instruction prompt using the entity relation dataset and append it to the end of the original text; Step 2.2: Use the forward propagation method of the pre-trained language model to encode the information in the entity relation dataset and obtain text representations.

4. The discriminative entity relation classification method based on a large language model representation as described in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Project the initial and final position vectors of the target entity through a multilayer perceptron to obtain boundary features; Step 3.2: Map the discrete type encoding of the entity to a continuous vector and fuse it with the boundary features; Step 3.3: Add the starting feature, ending feature and type vector together, and use the entity point-level representation obtained by formula (1); e Point <MLP start (h start )+MLP end (h end )+e type (1) Among them, h start h end These represent the start and end position vectors of the entity span, respectively; MLP start MLP end Represents an independent multilayer perceptron projection function; e type This represents the embedding vector obtained by encoding the entity type.

5. The discriminative entity relation classification method based on a large language model representation as described in claim 4, characterized in that: Step 3.1 is implemented as follows: Step 3.1.1: In the encoded sequence, determine the vector representations of the start and end positions corresponding to the target entity; Step 3.1.2: Project the vectors at the start and end positions through independent multilayer perceptrons to obtain boundary features.

Images