A method for extracting triples from large entity models based on knowledge injection

Abstract

This invention discloses a method for extracting triples from a large entity recognition model based on knowledge injection, comprising the following steps: obtaining input text and task prompts for the triples to be extracted; inputting the input text into a pre-trained large entity recognition model to obtain entity information of the input text, wherein the entity information includes the entity and entity position of the input text; inputting the input text, task prompts, and entity information into a pre-trained large triple extraction model to obtain the triple extraction result of the input text. This invention enhances the large model's capabilities in long text processing and detailed querying by injecting the output of the entity recognition model into the large model, while improving the model's generalization ability through specially designed data and a three-stage training strategy, reducing the illusion of entity extraction.

Description

Technical Field

[0001] This invention relates to the field of information extraction technology, and more specifically, to a method for extracting large model triples based on entity large model knowledge injection. Background Technology

[0002] In knowledge graph construction, a triple typically refers to a combination of two entities and a relation, represented as (entity 1, relation, entity 2). This structure can describe various semantic relationships between entities, such as "Einstein" - "born in" - "Germany". The triple is the basic unit for storing and representing knowledge in a knowledge graph. Traditionally, triple extraction using deep learning methods involves using transformer models or variants to identify entities in the input text and then predicting relationships based on entity locations. This method relies heavily on labeled data, has limited generalization ability, and struggles to adapt to new domains. With the rise of large-scale models, methods using large models for triple extraction effectively utilize pre-trained knowledge, reduce reliance on labeled data, and improve generalization. However, methods using large models for triple extraction face the following difficulties and challenges:

[0003] 1. Improve entity detection capabilities in long texts: This requires addressing the illusion problem that large models encounter when extracting triples from long texts, i.e., incorrectly extracting non-existent entities or missing entity recognition, and incorrectly generating non-existent entity relationships. This necessitates enhancing the entity detection capabilities of large models through knowledge injection from external entity models. One of the most critical problems faced by common comment filtering and analysis techniques when processing social media text is insufficient contextual understanding. Social media comments typically contain rich emotional content and complex contextual information, and often include omissions, typos, and creative spellings, all of which increase the difficulty of text understanding. Background techniques often rely on standardized text input and explicit grammatical structures, but the non-standardized nature of social media text makes these techniques difficult to adapt. Therefore, insufficient contextual understanding is a major challenge faced by the above techniques when processing social media comments.

[0004] 2. Integrating Entity Recognition Models: Integrating entity recognition models into larger models using different strategies to improve triple extraction performance is a technical challenge. Effective methods need to be found to allow larger models to utilize external models to enhance overall performance.

[0005] 3. Designing Domain-Adaptable Models: Traditional entity recognition models rely on domain-specific data, making it challenging to design models that can adapt to different domains and reduce error propagation. Many analytics techniques may perform well on specific types of data or within specific domains, but their performance can drop sharply when faced with new or diverse data. For example, a model trained on reviews of a specific product may struggle to analyze reviews of other products. The diversity and complexity of social media reviews require analytics techniques to have strong generalization capabilities, adapting to texts in different languages, topics, and contexts. However, many existing techniques tend to focus on specific types of text or specific domains, lacking sufficient generalization ability. This limits the application of these techniques in different scenarios and increases the difficulty of technology deployment and maintenance. Furthermore, considering the imperfections of entity recognition models, it is necessary to design appropriate training tasks and improve the model's accuracy and generalization ability through adversarial training.

[0006] 4. Text Diversity and Data Annotation: The diversity and complexity of text data, along with different text types, make triple extraction a challenging task. Furthermore, creating high-quality labeled datasets is a time-consuming and laborious task, especially for domain-specific or language-specific texts. A lack of sufficiently large and high-quality training data can limit model performance and generalization ability. In addition, the rapid changes in social media text mean that labeled data can quickly become outdated, requiring constant updates and maintenance. The quality of labeled data also affects the accuracy of analysis results. If the labeled data contains biases or errors, the trained model will inherit these biases, leading to inaccurate analysis results. Moreover, for comments on emerging topics or in specific contexts, it may be difficult to find enough labeled data for effective training. Therefore, the reliance on high-quality labeled data is another major challenge facing existing technologies. Domain adaptability is also a challenge in triple extraction; models may perform poorly in unseen domains, thus requiring solutions to the technical problem of effectively extracting triples from text data to construct knowledge graphs. Summary of the Invention

[0007] This invention provides a method for extracting large model triples based on entity large model knowledge injection, which solves the technical problems of existing technologies such as the illusion of large models and the weak ability to query details of long texts.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0009] This invention provides a method for extracting triples from large entity models based on knowledge injection, comprising the following steps:

[0010] Obtain the input text and task prompts for the triples to be extracted;

[0011] The input text is input into a pre-trained entity recognition model to obtain entity information of the input text, including the entity and entity location of the input text.

[0012] The input text, task prompts, and entity information are input into a pre-trained triplet extraction model to obtain the triplet extraction results of the input text.

[0013] This invention breaks down the ternary extraction task. Entity recognition is performed in the large entity recognition model, and then the entity location information is provided to the large ternary extraction model. With prior knowledge, the large ternary extraction model can focus more on judging entity types and relationships between entities, while retaining a certain ability to distinguish entity information. The accuracy of subsequent extraction will be higher, illusions will be reduced, and the technical problems of illusions in large models and weak ability to query details in long texts in existing technologies will be solved.

[0014] Furthermore, the training process for the pre-trained entity recognition model and the pre-trained triple extraction model includes:

[0015] Construct a first training set, which includes input text, triplet-extracted prompt words, and entity information;

[0016] The triplet extraction model is trained using the first training set to obtain a pre-trained triplet extraction model.

[0017] Construct a second training set, which includes the input text and entity extraction prompts;

[0018] The second training set is used to train a large entity recognition model, resulting in a pre-trained large entity recognition model.

[0019] The output of the pre-trained entity recognition model is connected to the input of the pre-trained triple extraction model, and the parameters of the pre-trained triple extraction model are frozen.

[0020] Construct a third training set, which includes input text, triplet-extracted prompt words, and entity information, and the data in the third training set is different from the data in the first training set.

[0021] The output of the pre-trained entity recognition model after training and connecting the third training set is combined with the pre-trained triplet extraction model to obtain the pre-trained entity recognition model.

[0022] Furthermore, the entity information in the first training set is processed as follows:

[0023] 50% of the entity information will be retained;

[0024] 30% of the entity information was incorrectly replaced, including the replacement of entity location information and the replacement of the entity itself;

[0025] Set 20% of the entity information to empty.

[0026] Furthermore, a large triplet extraction model is trained using the first training set to obtain a pre-trained large triplet extraction model, including:

[0027] The first LoRa module is introduced into the first preset large model, and a text embedding layer is added to obtain the triple extraction large model;

[0028] The input text and triplet-extracted prompt words from the first training set are directly input into the backbone network of the preset large model. The entity information from the first training set is converted into text embedding representation through the text embedding layer and then input into the backbone network of the preset large model for training. During training, the parameters of the backbone network of the preset large model are frozen, and only the parameters of the text embedding layer and LoRa module are fine-tuned.

[0029] During training, the first loss function is used to calculate the error between the predicted value and the true value of the current triplet under the large model parameters. Then, the gradient of the first loss function with respect to each parameter is calculated through the backpropagation algorithm.

[0030] Based on the calculated gradient and the preset learning rate, the parameters of the triplet extraction model are updated using gradient descent or its variant optimization algorithm until the first loss function of the triplet extraction model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained triplet extraction model.

[0031] Furthermore, the first loss function includes:

[0032] Text generation cross-entropy loss L triple_generation :

[0033]

[0034] In the formula, N represents the total number of samples, and y ij P represents the true probability that the triplet extraction model generates the j-th word at position i. ij This represents the predicted probability of the triplet extraction model generating the j-th word at position i.

[0035] Entity position output loss L triple_position :

[0036]

[0037] In the formula, m represents the number of entities. This indicates the actual position of the first character of the i-th entity. This represents the actual position of the last character of the i-th entity. This indicates the predicted position of the first character of the i-th entity. w1 and w2 represent the predicted position of the last character of the i-th entity, and w1 and w2 represent the weight parameters.

[0038] Entity quantity loss L triple_quantity :

[0039] L triple_quantity =log(1+|PT|)

[0040] In the formula, P is the predicted number of entities, and T is the actual number of entities;

[0041] First loss function L triplr_total for:

[0042]

[0043] In the formula, λ1, λ2 and λ3 are hyperparameters.

[0044] Furthermore, the entity recognition large model is trained using the second training set to obtain a preliminarily trained entity recognition large model, including:

[0045] The second pre-set large model is used as the entity recognition large model;

[0046] During training, all parameters of the model will be updated during fine-tuning. First, the error between the predicted value and the true value under the current model parameters is calculated using the second loss function. Then, the gradient of the second loss function with respect to each parameter is calculated using the backpropagation algorithm.

[0047] Parameter update: Based on the calculated gradient and the preset learning rate, the parameters of the model are updated using gradient descent or its variant optimization algorithm until the loss function of the entity recognition large model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained entity recognition large model.

[0048] Furthermore, the number of parameters in the preset second largest model is less than the number of parameters in the preset first largest model.

[0049] Furthermore, the second loss function L ner_generation ,include:

[0050]

[0051] Furthermore, by using the output of the pre-trained entity recognition model trained and connected using the third training set, and the pre-trained triplet extraction model, a pre-trained entity recognition model is obtained, including:

[0052] A second LoRa module is introduced into the pre-trained entity recognition model, and only the parameters of the second LoRa module are updated during the training process;

[0053] The input text in the third training set is processed by the pre-trained entity recognition model to generate entity recognition results.

[0054] The generated entity recognition results, the input text in the third training set, and the triplet extraction prompts in the third training set are used as inputs to the pre-trained triplet extraction model to obtain the output of the pre-trained triplet extraction model.

[0055] The third loss function is calculated by extracting the output of the large model from the pre-trained triplet;

[0056] The parameters of the second LoRa module are updated based on the gradient of the third loss function.

[0057] Furthermore, the third loss function includes:

[0058]

[0059] In the formula, L ner_quantity This represents the loss generated.

[0060] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0061] This invention proposes a method to enhance the triple extraction performance of a large model through entity model knowledge injection. By injecting the output of an entity recognition model into a large model, this invention enhances the model's capabilities in long text processing and detailed querying. Simultaneously, through specially designed data and a three-stage training strategy, it improves the model's generalization ability and reduces the illusion of entity extraction. This method not only improves the accuracy of triple extraction but also enhances the model's adaptability across different domains and text types, providing an effective technical means for knowledge graph construction. Attached Figure Description

[0062] Figure 1 A flowchart illustrating a method for extracting large model triples based on entity large model knowledge injection, provided in an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram illustrating the use of the first training set to train a large model for triple extraction, as provided in an embodiment of the present invention.

[0064] Figure 3 This is a schematic diagram of the output of the pre-trained entity recognition large model after training and connecting using the third training set, and the pre-trained triplet extraction large model provided in an embodiment of the present invention.

[0065] Figure 4 This is a schematic diagram of the enhanced large model triple extraction process provided in an embodiment of the present invention. Detailed Implementation

[0066] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0067] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0068] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0069] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0070] Example 1

[0071] This invention provides a method for extracting large model triples based on entity large model knowledge injection, such as... Figure 1 As shown, it includes the following steps:

[0072] Obtain the input text and task prompts for the triples to be extracted;

[0073] The input text is input into a pre-trained entity recognition model to obtain entity information of the input text, including the entity and entity location of the input text.

[0074] The input text, task prompts, and entity information are input into a pre-trained triplet extraction model to obtain the triplet extraction results of the input text.

[0075] This invention breaks down the ternary extraction task. Entity recognition is performed in the large entity recognition model, and then the entity location information is provided to the large ternary extraction model. With prior knowledge, the large ternary extraction model can focus more on judging entity types and relationships between entities, while retaining a certain ability to distinguish entity information. The accuracy of subsequent extraction will be higher, illusions will be reduced, and the technical problems of illusions in large models and weak ability to query details in long texts in existing technologies will be solved.

[0076] Example 2

[0077] This embodiment, based on Embodiment 1, continues to disclose the following content:

[0078] The training process for the pre-trained entity recognition model and the pre-trained triple extraction model includes:

[0079] Construct a first training set, which includes input text, triplet-extracted prompts, and entity information;

[0080] The triplet extraction model is trained using the first training set to obtain a pre-trained triplet extraction model.

[0081] Construct a second training set, which includes the input text and entity extraction prompts;

[0082] The second training set is used to train a large entity recognition model, resulting in a pre-trained large entity recognition model.

[0083] The output of the pre-trained entity recognition model is connected to the input of the pre-trained triple extraction model, and the parameters of the pre-trained triple extraction model are frozen.

[0084] Construct a third training set, which includes input text, triplet-extracted prompt words, and entity information, and the data in the third training set is different from the data in the first training set;

[0085] The output of the pre-trained entity recognition model after training and connecting the third training set is combined with the pre-trained triplet extraction model to obtain the pre-trained entity recognition model.

[0086] In a specific embodiment, the entire training process includes three stages:

[0087] The training objective of the first stage is to enable the triple extraction model to learn to extract triples using the text index information output by the entity recognition model. After the entity recognition model is trained, it can output the location of entities in the text and text fragments, but the triple extraction model cannot directly utilize this information. Therefore, the model architecture needs to be modified and corresponding tasks need to be set to enhance it.

[0088] The training objective of the second stage is to train a large entity recognition model that outputs the location of entities in the text and text fragments, which will be used to complete the entity recognition task later.

[0089] The training objective of the third stage is to assemble the two large models, freeze the triplet to extract the parameters of the large model, and only fine-tune the parameters of the entity recognition large model. This leverages the capabilities of the large model with large parameters to enhance the ability of the entity recognition large model, further reducing the illusion problem in the generation process of the small model.

[0090] Example 3

[0091] Based on Example 2, this embodiment continues to disclose the following content:

[0092] Furthermore, the entity information in the first training set is processed as follows:

[0093] 50% of the entity information is retained, including the correct entity location information when annotating;

[0094] 30% of the entity information was incorrectly replaced, including the replacement of entity location information and the replacement of the entity itself, such as:

[0095] Location information replacement:

[0096] Input: "end_index":16,"start_index":1,"text":"Cartier"

[0097] Replace with: "end_index":18,"start_index":4,"text":"Cartier"

[0098] This involves replacing entity content with data that is not present in the text.

[0099] Input: "end_index":16,"start_index":1,"text":"Dia"

[0100] Set 20% of the entity information to empty.

[0101] In this embodiment, because even a well-trained entity recognition model cannot guarantee 100% accuracy in identifying the position of entities within text fragments, this embodiment modifies the manually labeled multi-domain triple extraction training set to avoid the triple extraction model relying too heavily on entity position information and to improve the model's generalization and fault tolerance. Appropriate proportions of entity position information are modified and replaced. Through this data setting method, the model has a certain degree of fault tolerance when the input position information or generated entities are incorrect, thus reducing the model's 100% dependence on entity position information.

[0102] Furthermore, a large triplet extraction model is trained using the first training set to obtain a pre-trained large triplet extraction model, including:

[0103] The first LoRa module is introduced into the first preset large model, and a text embedding layer is added to obtain the triple extraction large model;

[0104] The input text and triplet-extracted prompt words from the first training set are directly input into the backbone network of the preset large model. The entity information from the first training set is converted into text embedding representation through the text embedding layer and then input into the backbone network of the preset large model for training. During training, the parameters of the backbone network of the preset large model are frozen, and only the parameters of the text embedding layer and LoRa module are fine-tuned.

[0105] During training, the first loss function is used to calculate the error between the predicted value and the true value of the current triplet under the large model parameters. Then, the gradient of the first loss function with respect to each parameter is calculated through the backpropagation algorithm.

[0106] Based on the calculated gradient and the preset learning rate, the parameters of the triplet extraction model are updated using gradient descent or its variant optimization algorithm until the first loss function of the triplet extraction model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained triplet extraction model.

[0107] In a specific implementation, during the first stage of training, the model is as follows: Figure 2 As shown, according to Figure 2 The model architecture shown is a combination model. The large model with large parameters that performs triple extraction is denoted as LLM_triple. Based on LLM_triple (such as llama3-70b), the LoRa (Low-Rank Adaptation) module is introduced for fine-tuning.

[0108] Figure 2 The function of the text embedding layer in the middle is:

[0109] For each training data point, the JSON string containing the two entity positions is converted into an embedding representation that the model can understand through an embedding layer. This step converts the entity's index information in the text into a numerical form.

[0110] In this embodiment, the format of the first training set is as follows:

[0111] The first training set includes the input text, triplet-based prompts, and entity information:

[0112] Input text: "Turns out the only thing separating me from Andy Lau is a Cartier watch hahaha #This move was a sure thing #Andy Lau #watch";

[0113] Entity information: "[{"end_index":16,"start_index":13,"text":"Cartier"},{"end_index":32,"start_index":29,"text":"Andy Lau"},{"end_index":36,"start_index":34,"text":"Watch"}]";

[0114] Prompt words:

[0115] ##Role

[0116] You are a social media knowledge graph analysis expert. You are highly skilled at extracting and identifying specific entities from text and determining the relationships between entities. Please strictly follow the task requirements to extract entities from the text and determine the relationships between them:

[0117] ##Entity Definition

[0118] 1. <Personal Name>: A personal name is a name used to identify and distinguish an individual. In different cultures and languages, personal names may include surnames (family names), given names (given names), middle names, nicknames, titles, etc. Personal names are usually used to refer to specific people, which can be historical figures, public figures, fictional characters, or any individual.

[0119] 2. <Organization Name>: Refers to a group of individuals, which can be a company, government agency, non-profit organization, school, etc. ...

[0121] ## Relationship Definition:

[0122] 1. <Parents>: The text often uses expressions such as "...father / mother", "as the parent of", "one of the parents of" or similar expressions.

[0123] 2. <Children>: The text often uses expressions such as "is the son / daughter of...", "is the child of...", or "is one of the descendants of..." ...

[0125] ### Labeling Principles:

[0126] 1. **Entity Extraction**: Carefully read the input text and extract all entities in the text that match the definition according to the "entity definition".

[0127] 2. **Entity Relationship Classification:** Carefully analyze the context of each entity, extract entity pairs that match the "Entity Relationship Definition," and output the relationship type between the entities.

[0128] 3. **Formatting Requirements:** The triple should include the subject (person, organization, or object), the relation, and the object (person, place, or object).

[0129] ##Articles to be extracted

[0130] {input_text}

[0131] ### Output format:

[0132] Please strictly follow the following format for output. If there are no entities, output an empty list for the entity results. If there are no entity relations, output an empty list for the triple results. The specific answer format is as follows:

[0133] Entity result: [{"end_index": end index,"start_index": start index,"text":"Andy Lau",},...{"end_index": end index,"start_index": start index,"text":"Cartier"}]

[0134] Triple result: [{"ent1":"Entity 1","relation":"Relation 1","ent2":"Entity 2"},...{"ent1":"Entity 3","relation":"Relation 2","ent2":"Entity 4"}]

[0135] Note: {input_text} will be filled with the input text. The ellipsis in the definition indicates that there are other definitions that are not fully written. This example only shows one example.

[0136] Given the above input, the large model output is the following string:

[0137] Entity: [{"end_index":16,"start_index":13,"text":"Cartier"},{"end_index":32,"start_index":29,"text":"Andy Lau"},{"end_index":36,"start_index":34,"text":"Watch"}],

[0138] Entity Relationship: [{"ent1":"Watch","relation":"Brand","ent2":"Cartier"},{"ent1":"Cartier","relation":"Endorser","ent2":"Andy Lau"}]

[0139] In a specific embodiment, a task generation method is used to complete the triplet extraction task, namely, input text + triplet extraction prompt words.

[0140] The input text and triplet-extracted cue words are used to construct triplet cue words, which are then directly input into the LLM_triple backbone network. This allows LLM_triple to simultaneously receive both the original text features and entity location features. The model is then fine-tuned using LoRa with the training set of triplets.

[0141] During training, the parameters of the LLM_tripled backbone network are frozen (i.e., the model parameters are not adjusted during training to avoid losing the capabilities learned by the model during pre-training). During fine-tuning training, only the parameters of the text embedding layer and the large model LoRa layer are fine-tuned.

[0142] Loss function design:

[0143] Text generation cross-entropy loss L triple_generation The focus is on the difference between the text generated by the model and the target text. For batch data, the cross-entropy loss is typically the average loss per sample, where N is the total number of data points in each batch, and y... ij P represents the true probability that the triplet extraction model generates the j-th word at position i. ij This represents the predicted probability of the triplet extraction model generating the j-th word at position i:

[0144]

[0145] In the formula, N represents the total number of samples, and y ij P represents the true probability that the triplet extraction model generates the j-th word at position i. ij This represents the predicted probability of the triplet extraction model generating the j-th word at position i.

[0146] Entity Position Loss (L) triple_position This embodiment requires a loss function to measure the accuracy of entity location indexing. This can be achieved by calculating the difference between the entity start and end indices predicted by the model and the actual indices:

[0147]

[0148] In the formula, m represents the number of entities. This indicates the actual position of the first character of the i-th entity. This represents the actual position of the last character of the i-th entity. This indicates the predicted position of the first character of the i-th entity. This represents the predicted position of the last character of the i-th entity. w1 and w2 represent weight parameters used to balance the importance of the start and end positions. These weights can be adjusted according to the actual needs of the production environment. In this embodiment, w1 = 0.6 and w2 = 0.4 are chosen because determining the start index of an entity is more important than determining its end index. This total loss function accumulates the positional losses of all entities, thus providing a measure of the overall positional prediction performance of the model.

[0149] Entity quantity loss L triple_quantity Error is calculated by taking the logarithm of the ratio of the predicted value to the actual value. This method is suitable when both the predicted and actual values ​​are positive.

[0150] L triple_quantity =log(1+|PT|)

[0151] In the formula, P is the predicted number of entities, and T is the actual number of entities;

[0152] The overall loss function combines the above loss functions to form a single overall loss function, which simultaneously optimizes entity recognition, location prediction, and entity number alignment. The first loss function L... triple_total It can be defined as:

[0153]

[0154] In the formula, λ1, λ2, and λ3 are hyperparameters used to balance the importance of different loss terms. In this embodiment, the values ​​are λ1 = 0.5, λ2 = 0.3, and λ3 = 0.2. The specific values ​​can be adjusted appropriately according to the production task. This loss function can be used for optimization during the training process. By minimizing this loss function, the model will pay more attention to the location index and number of entities, in addition to the accuracy of text generation.

[0155] Furthermore, the entity recognition large model is trained using the second training set to obtain a preliminarily trained entity recognition large model, including:

[0156] The second pre-set large model is used as the entity recognition large model;

[0157] During training, all parameters of the model will be updated during fine-tuning. First, the error between the predicted value and the true value under the current model parameters is calculated using the second loss function. Then, the gradient of the second loss function with respect to each parameter is calculated using the backpropagation algorithm.

[0158] Parameter update: Based on the calculated gradient and the preset learning rate, the parameters of the model are updated using gradient descent or its variant optimization algorithm until the loss function of the entity recognition large model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained entity recognition large model.

[0159] In a specific embodiment, the second training set includes input text and entity extraction prompts:

[0160] Input text: "Turns out the only thing separating me from Andy Lau is a Cartier watch hahaha #This move was a sure thing #Andy Lau #watch";

[0161] Entity extraction prompts:

[0162] ##Role

[0163] You are an expert in the field of social media discourse analysis. You are very good at reading social media texts and extracting useful information from them. Next, I will give you an input text. Please extract entities related to the beauty industry according to the following requirements.

[0164] ###Task Explanation:

[0165] Entity recognition, or naming entity recognition, requires extracting entities with specified dimensions from the original text.

[0166] Dimension: This refers to the category of an entity; an entity can belong to multiple categories simultaneously.

[0167] There are N types of dimensions, namely: brand, category, product name...

[0168] ###Dimension Definition:

[0169] 1. <Brand>: Brands of beauty products (makeup, skincare, beauty devices, perfumes, and related tools) and daily chemical products (daily chemical products), including both Chinese and English names, such as: Saint Laurent, MAC;

[0170] 2. <Category>: Beauty products (makeup, skincare, daily chemical products, etc., such as lipsticks, face masks, hair removal devices, etc.); ...

[0172] ### Labeling Principles:

[0173] 1. **Precise Extraction**: If no entity / phrase matches the given dimensions appears in the article, no other entities will be extracted.

[0174] 2. **Complete Extraction**: Entities in certain dimensions may be phrases rather than simple nouns; ...

[0176] ### Enter text

[0177] {input_text}

[0178] ### Output format:

[0179] Please strictly follow the following format for output. If there is no entity, output an empty list of the entity results. The specific answer format is as follows:

[0180] Entity result: [{"end_index": end index,"start_index": start index,"text":"entity","type":"entity type"},...{"end_index": end index,"start_index": start index,,"text":"entity","type":"entity type"}).

[0181] Corresponding large model output:

[0182] [{"end_index":6,"start_index":3,"text":"Andy Lau","type":"Name"},{"end_index":16,"start_index":13,"text":"Cartier","type":"Brand"},{"end_index":36,"start_index":34,"text":"Watch","type":"Category"}]]

[0183] In the above large model output, text represents the entity, start_index represents the position of the first character of the entity in the sentence, end_index is the index of the last character of the entity in the sentence, and type represents the type of the entity.

[0184] Furthermore, the number of parameters in the preset second largest model is less than the number of parameters in the preset first largest model.

[0185] In a specific implementation, a small-parameter large model, denoted as LLM_ner, is fine-tuned in advance using a multi-domain entity recognition training set. The model can be a large model with fewer than 10b parameters (b represents tens of billions of parameters), such as gemma-2b, minicpm-4b, or qwen2.5-1.5b. The purpose of choosing a small-parameter large model is because the inference speed of the model is proportional to the number of parameters. Since speed is required in production environments, and the small-parameter large model only needs to perform one entity recognition task, it does not need a model with such complex parameters.

[0186] Fine-tuning strategy selection: The SFT (full parameter fine-tuning) strategy is adopted, in which all parameters of the model will be updated during the fine-tuning process. The loss function used is the same as that used in the previous triplet training task.

[0187]

[0188] Furthermore, by using the output of the pre-trained entity recognition model trained and connected using the third training set, and the pre-trained triplet extraction model, a pre-trained entity recognition model is obtained, including:

[0189] A second LoRa module is introduced into the pre-trained entity recognition model, and only the parameters of the second LoRa module are updated during the training process;

[0190] The input text in the third training set is processed by the pre-trained entity recognition model to generate entity recognition results.

[0191] The generated entity recognition results, the input text in the third training set, and the triplet extraction prompts in the third training set are used as inputs to the pre-trained triplet extraction model to obtain the output of the pre-trained triplet extraction model.

[0192] The third loss function is calculated by extracting the output of the large model from the pre-trained triplet;

[0193] The parameters of the second LoRa module are updated based on the gradient of the third loss function.

[0194] In a specific embodiment, according to Figure 3 The model architecture shown is a combination of three parts: a large entity recognition model, a text embedding layer, and a large triple extraction model. The parameters that need to be frozen include the already fine-tuned main parameters of the large model `LLM_triple`, the parameters of the text embedding layer, the main parameters of `LLM_ner`, and the LoRa module of `LLM_triple`. The parameters that need to be updated include those for the LoRa layer of `LLM_ner`.

[0195] Reuse the loss from previous triple training + the generation loss from the large entity recognition model LLM_ner:

[0196]

[0197] In the formula, L ner_quantity This represents the loss generated.

[0198] After completing the above steps, the entity recognition model and the triple extraction model will work together better, improving the overall system performance.

[0199] After training with data from multiple domains, inference with subsequent new data, such as... Figure 4 As shown, each piece of data first goes through LLM_ner to obtain the entity index string, then goes through the text embedding layer to be converted into entity location information that the large model can understand, and then combines the task prompt words into the triple extraction model to obtain the triple result. The specific process is shown in the figure below. Since the triple extraction model can read the entity index information, the entity recognition model can be replaced by entity recognition models in other fields, which has a certain degree of universality.

[0200] The same or similar labels correspond to the same or similar parts;

[0201] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0202] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting triples from large entity models based on knowledge injection, characterized in that, Includes the following steps: Obtain the input text and task prompts for the triples to be extracted; The input text is input into a pre-trained entity recognition model to obtain entity information of the input text, including the entity and entity location of the input text. The input text, task prompts, and entity information are input into a pre-trained triplet extraction model to obtain the triplet extraction result of the input text. The training process for the pre-trained entity recognition model and the pre-trained triple extraction model includes: Construct a first training set, which includes input text, triplet-extracted prompt words, and entity information; The triplet extraction model is trained using the first training set to obtain a pre-trained triplet extraction model. Construct a second training set, which includes the input text and entity extraction prompts; The second training set is used to train a large entity recognition model, resulting in a pre-trained large entity recognition model. The output of the pre-trained entity recognition model is connected to the input of the pre-trained triple extraction model, and the parameters of the pre-trained triple extraction model are frozen. Construct a third training set, which includes input text, triplet-extracted prompt words, and entity information, and the data in the third training set is different from the data in the first training set. The output of the pre-trained entity recognition model after training and connection using the third training set is combined with the pre-trained triplet extraction model to obtain the pre-trained entity recognition model. The triplet extraction model is trained using the first training set to obtain a pre-trained triplet extraction model, including: The first LoRa module is introduced into the first preset large model, and a text embedding layer is added to obtain the triple extraction large model; The input text and triplet-extracted prompt words from the first training set are directly input into the backbone network of the preset large model. The entity information from the first training set is converted into text embedding representation through the text embedding layer and then input into the backbone network of the preset large model for training. During training, the parameters of the backbone network of the preset large model are frozen, and only the parameters of the text embedding layer and LoRa module are fine-tuned. During training, the first loss function is used to calculate the error between the predicted value and the true value of the current triplet under the large model parameters. Then, the gradient of the first loss function with respect to each parameter is calculated through the backpropagation algorithm. Based on the calculated gradient and the preset learning rate, the parameters of the triplet extraction model are updated using gradient descent or its variant optimization algorithm until the first loss function of the triplet extraction model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained triplet extraction model.

2. The method for extracting large model triples based on entity large model knowledge injection according to claim 1, characterized in that, The entity information in the first training set is processed as follows: 50% of the entity information will be retained; 30% of the entity information was incorrectly replaced, including the replacement of entity location information and the replacement of the entity itself; Set 20% of the entity information to empty.

3. The method for extracting large model triples based on entity large model knowledge injection according to claim 2, characterized in that, The first loss function includes: Text generation cross-entropy loss L triple_generation : In the formula, N represents the total number of samples, and y ij P represents the true probability that the triplet extraction model generates the j-th word at position i. ij This represents the predicted probability of the triplet extraction model generating the j-th word at position i. Entity position output loss L triple_position : In the formula, m represents the number of entities. This indicates the actual position of the first character of the i-th entity. This represents the actual position of the last character of the i-th entity. This indicates the predicted position of the first character of the i-th entity. w1 and w2 represent the predicted position of the last character of the i-th entity, and w1 and w2 represent the weight parameters. Entity quantity loss L triple_quantity : L triple_quantity =log(1+|P-T|) In the formula, P is the predicted number of entities, and T is the actual number of entities; First loss function L triple_total for: In the formula, λ1, λ2 and λ3 are hyperparameters.

4. The method for extracting large model triples based on entity large model knowledge injection according to claim 3, characterized in that, The entity recognition large model is trained using the second training set to obtain a preliminarily trained entity recognition large model, including: The second pre-set large model is used as the entity recognition large model; During training, all parameters of the model are updated during fine-tuning. First, the error between the predicted value and the true value under the current model parameters is calculated using the second loss function. Then, the gradient of the second loss function with respect to each parameter is calculated using the backpropagation algorithm. Parameter update: Based on the calculated gradient and the preset learning rate, the parameters of the model are updated using gradient descent or its variant optimization algorithm until the loss function of the entity recognition large model is less than the preset value or the preset number of iterations is reached, thus obtaining the pre-trained entity recognition large model.

5. The method for extracting large model triples based on entity large model knowledge injection according to claim 4, characterized in that, The number of parameters in the preset second largest model is less than the number of parameters in the preset first largest model.

6. The method for extracting large model triples based on entity large model knowledge injection according to claim 4, characterized in that, The second loss function L ner_generation ,include:

7. The method for extracting large model triples based on entity large model knowledge injection according to any one of claims 4 to 6, characterized in that, Using the output of the pre-trained entity recognition model trained on the third training set and the pre-trained triplet extraction model, a pre-trained entity recognition model is obtained, including: A second LoRa module is introduced into the pre-trained entity recognition model, and only the parameters of the second LoRa module are updated during the training process; The input text in the third training set is processed by the pre-trained entity recognition model to generate entity recognition results. The generated entity recognition results, the input text in the third training set, and the triplet extraction prompts in the third training set are used as inputs to the pre-trained triplet extraction model to obtain the output of the pre-trained triplet extraction model. The third loss function is calculated by extracting the output of the large model from the pre-trained triplet; The parameters of the second LoRa module are updated based on the gradient of the third loss function.

8. The method for extracting large model triples based on entity large model knowledge injection according to claim 7, characterized in that, The third loss function includes: In the formula, L ner_quantity This represents the loss generated.

Images