Construction method and system of cigarette knowledge base based on large language model and knowledge graph

Abstract

A method and system for constructing a cigarette knowledge base based on a large language model and knowledge graph are disclosed. After collecting and labeling data, a pre-trained model is obtained by fine-tuning the large language model using LoRA. This model generates a knowledge graph structure based on prompt words derived from natural language text, which is then used to generate corresponding Cypher statements and import them into a database. Based on a small-scale data sample in the cigarette field, this invention trains a model capable of automatically generating knowledge graph structures from natural language text using the prompt word method and LoRA fine-tuning technology. The model is then stored in a graph relation database, enabling entity relation extraction from cigarette-related natural language segments and the automated construction of a knowledge graph related to cigarette entities.

Description

Technical Field

[0001] This invention relates to a technology in the field of natural language processing, specifically a method and system for constructing a cigarette knowledge base based on a large language model and knowledge graph. Background Technology

[0002] In the cigarette industry, there exists a wealth of textual information rich in semantic and entity content, containing a wealth of tobacco knowledge relevant to cigarette retail scenarios. For example, the following passage, "XXX is a flue-cured cigarette, attracting many consumers with its unique design and packaging. The main color of this cigarette box is pink, complemented by a magenta secondary color, giving it a gentle and graceful feel," contains valuable product information about the type of cigarette XXX and the color of its box. Before the rapid development and large-scale application of natural language processing technology, the level of digitalization and informatization in the cigarette retail sector was low. The construction of traditional cigarette knowledge bases mainly relied on the collection, analysis, and extraction of cigarette knowledge by experts in the field. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies, such as the lack of semantic representation data and training methods in the tobacco vertical field, and the lack of modeling of common-sense conceptual knowledge in the tobacco field. It proposes a method and system for constructing a cigarette knowledge base based on a large language model and knowledge graph. Based on a small-scale data sample in the cigarette field, the invention trains a model that can automatically generate knowledge graph structures from natural language text through the prompt word method and LoRA fine-tuning technology, and stores it in a graph relation database. This enables the extraction of entity relations from cigarette-related natural language segments and the automatic construction of a knowledge graph related to cigarette entities.

[0004] This invention is achieved through the following technical solution:

[0005] This invention discloses a method for constructing a cigarette knowledge base based on a large language model and a knowledge graph. After collecting and labeling data, the large language model is fine-tuned using LoRA to obtain a pre-trained model. This model generates a graph structure based on prompt words constructed from natural language text, which is then used to generate corresponding Cypher statements and import them into the database.

[0006] The large language model mentioned is Meta-llama3-8B-Instruct, which uses training set data for LoRA fine-tuning.

[0007] The training set data for LoRA fine-tuning comes from cigarette-related text data collected in a multimodal manner. The collection methods include, but are not limited to, web scraping, recording of conversations at offline cigarette retail stores, and descriptions in official cigarette documentation.

[0008] The prompt words are generated quickly using langchain's FewShotPromptTemplete, and the few-shot method is used to enhance the prompt words. The constructed prompt task includes task labeling, few-shot samples (annotated original text and expected output), and user input text.

[0009] The pre-trained model constructs a task based on the knowledge graph structure, extracting the knowledge graph structure from the input text. For example, given the text "XXX is a flue-cured cigarette, which attracts many consumers with its unique appearance design and packaging. The main color of this cigarette box is pink, with a secondary color of rose red, giving people a gentle and beautiful feeling," the system can identify the knowledge graph structure of (label:`Tobacco Product`{name:"XXX"})-[label:`Type`]-(label:`Type`{name:"Flue-cured"}).

[0010] The graph relation database mentioned is the neo4j graph database, which stores relevant knowledge graph data using a graph data model. Technical effect

[0011] This invention employs a common-sense graph modeling method for tobacco retail scenarios and a TextToCypher method based on prompt words in the tobacco vertical domain (including formatting, prompt generation, and entity disambiguation in the tobacco retail domain). Compared to existing technologies, this invention enables automated graph structure extraction and knowledge graph structure construction. Based on the constructed knowledge base, it can complete customer knowledge question answering tasks in tobacco retail scenarios. In terms of performance, tested on a self-built test set, compared to existing graph structure extraction and construction technologies, this invention achieves an accuracy of 93.8% in the graph structure extraction task, a 16.7% improvement over general models, and an F1 score of 91.669, a 24.2% improvement over general models. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention;

[0013] Figure 2 This is a schematic diagram of the system modules in the embodiment;

[0014] Figure 3 A knowledge graph structure diagram constructed to model common-sense concepts. Detailed Implementation

[0015] like Figure 2As shown in this embodiment, it is a cigarette knowledge base construction system based on a large language model and knowledge graph. It includes: an instruction construction and generation module, a large model knowledge relationship extraction module, a cypher statement generation module, and a graph construction and disambiguation module. Specifically: the instruction construction and generation module generates corresponding instructions based on the natural language text segments input into the system, inputs the constructed instruction content into the large model, and the trained model returns the extracted knowledge triples for the text segments, along with the corresponding graph structure. The cypher statement generation module further processes the graph structure returned by the model, constructing corresponding cypher statements from the graph structure, and then passes the cypher statements to the graph construction and disambiguation module. The graph construction and disambiguation module performs disambiguation processing on the entity content designed for the cypher statement in the current graph structure. If there is no triple structure for the cypher statement to be inserted in the current graph structure, then the current knowledge triple structure is inserted into the knowledge graph.

[0016] like Figure 1 As shown, this embodiment illustrates the method for constructing a cigarette knowledge base based on the aforementioned system, including:

[0017] Step 1) Collect raw data through a multi-source modal approach, including: web crawling, recording of conversations with offline cigarette retail stores, and descriptions from official cigarette documentation.

[0018] Step 2) Model common-sense conceptual knowledge graphs for knowledge in the tobacco retail scenario, specifically including:

[0019] 2.1) For the task of constructing a cigarette knowledge base, a knowledge graph will be established that covers as much relevant knowledge as possible in the cigarette retail scenario. Specifically, a common-sense concept knowledge graph will be built first to assist the model in understanding natural language text. In the concept graph, the graph structure will be built layer by layer from top to bottom, according to the categories of natural text semantic understanding.

[0020] The knowledge understanding mentioned includes:

[0021] a) What is: This dimension represents the conceptual affiliation of entities in knowledge. Taking tobacco products in the cigarette industry as an example, "XXX" is a type of tobacco product, "flue-cured" indicates the category of tobacco, and "fragrant" indicates the flavor characteristics of cigarettes.

[0022] b) What: This dimension represents the attribute relationship between different concepts in knowledge. Under this dimension, one entity is indicated as an attribute of another entity. For example, in the structure “XXX”-“Type”-“Flue-cured tobacco type”, “Flue-cured tobacco type” is the specific attribute of the cigarette “XXX” in terms of the concept of “type”.

[0023] c) Association: This dimension represents the association between entities in knowledge. Under this dimension, different entities do not have a hierarchical relationship, but only represent a connection at a certain conceptual level. For example, in the structure "XXX"-"Competitor"-"YYY", the tobacco product "XXX" and the tobacco product "YYY" have a "competitor" relationship. This relationship is only an association at a certain level and does not indicate the hierarchical classification of the entities.

[0024] 2.2) Based on the above three dimensions, a top-down graph structure will be constructed. Two types of nodes and three types of relationships will be defined in the graph structure, specifically including:

[0025] Top-level concept nodes: These nodes represent the top-level concept nodes in the graph structure. Each node represents a specific entity category. Top-level concept nodes can be divided into two categories: one is specific entities in the cigarette retail scenario, such as "tobacco products" and "manufacturers"; the other is specific attributes for specific entities, such as "type" and "flavor". The construction of top-level concept nodes relies on the analysis of tobacco specification information, which is provided by tobacco manufacturers and obtained from tobacco product-related websites. By analyzing and statistically analyzing the frequency data of relevant specification information mentioned in customer conversations in offline stores, the ten specifications that users are most concerned about are obtained, and the top-level concept nodes are built based on this specification information.

[0026] Atomic entity nodes: Atomic entity nodes are the smallest indivisible entity nodes in the tobacco domain. An atomic entity node must have its own independent semantics and can only be attached to a single top-level concept node. For example, "XXX" is an atomic entity node for "tobacco products".

[0027] Three relationships are defined based on these two types of nodes:

[0028] Inheritance relationship: Inheritance relationship represents the hierarchical relationship between nodes in semantics. It is represented by "isA" in the graph, that is, a node is a subclass of another node. Inheritance relationship only exists between atomic entity nodes and top-level concept nodes, such as "XXX"-"isA"-"Tobacco Products".

[0029] Attribute relationships: Attribute relationships indicate that one node describes the attribute characteristics of another node. In the graph, this is represented by "is", such as "XXX" - "is" - "Flue-cured tobacco type".

[0030] Association: A link relationship indicates that there is an association between one node and another node, and this relationship is defined by a specific event. For example, in the tobacco retail scenario, if one type of cigarette is a competitor of another type of cigarette, this relationship is an association relationship, represented as "XXX" - "competitor" - "YYY".

[0031] Based on the above definitions, we explore the knowledge graph structure in the tobacco context. For example... Figure 3 The diagram shows a portion of the knowledge graph structure in a tobacco retail scenario, including entity types, relation types, and tag definitions.

[0032] Step 3) The collected data is labeled with questions and answers, and the labeled data is used for LoRA fine-tuning to enhance the model's question-answering ability in the tobacco vertical field.

[0033] The aforementioned question-and-answer annotation includes: annotating dialogue entities and dialogue content.

[0034] An example with annotations in this embodiment is shown below:

[0035] {

[0036] “conversations”:[

[0037] {

[0038] "from": "human",

[0039] "value": "What distinctive flue-cured cigarettes has ZZZ Company produced?"

[0040] }

[0041] {

[0042] "from":"model",

[0043] "value": "XXX is a flue-cured cigarette that attracts many consumers with its unique design and packaging. The main color of the cigarette box is pink, with a secondary color of rose red, giving it a gentle and graceful feel."

[0044] } ]

[0046] }

[0047] Step 4) Perform formatting operations before model extraction.

[0048] The formatting process described above involves formatting the user's natural language text to facilitate subsequent regular expression matching. If the natural language text contains parentheses such as "(", ")", or other special identifiers, these will be uniformly removed during the formatting stage; otherwise, the entity reference "XXX" in the natural language text will not match the entity reference "XXX" recorded in the neo4j database.

[0049] Step 5) Construct large language model prompt words using the FewShotPromptTemplete component from the langchain component library to realize the transformation from natural language to knowledge graph structure, specifically including:

[0050] 5.1) Perform graph structure annotation on natural language text to serve as the expected response of the large language model in the few-shot samples.

[0051] The graph structure annotation includes: for entity nodes in the graph, it is labeled in the form of (label:`**`{name:”@@”}), where ** is the entity label to be extracted and @@ is the entity node to be extracted; for relationships in the graph, it is labeled in the form of [label:`**`], where ** is the relationship label to be extracted. The relationship category is set by experts in the cigarette industry based on the cigarette specification dimension. The specific relationship dimensions are shown in Table 1.

[0052] Table 1

[0053] 5.2) The prompt task in the prefix task is as follows: "You are an expert in the cigarette knowledge domain. Please extract the knowledge graph structure of cigarette entities and relations in the following text. The extracted graph triple structure is identified by (label:`**`{name:"@@"})-[label:`**`]-(label:`**`{name:"@@"}), and the extracted individual entity is identified by (label:`**`{name:"@@"}), where ** is the label of the entity or relation you have identified, and @@ is the name of the entity or relation you have identified."

[0054] 5.3) Combine the labeled data with the text segments to form a few-shot sample.

[0055] The template for the few-shot sample is:

[0056] examples = [

[0057] {

[0058] "query": "[cigarette-related text]",

[0059] "answer": "[The manually annotated corresponding answer]"

[0060] } ]

[0062] 5.4) Read the natural language text content corresponding to the user input.

[0063] A specific example of the natural language text content is: "AAA is a new slim cigarette with flavor capsules created by ZZZ as an 'expert in the fusion of tobacco and tea'."

[0064] 5.5) Call the template of the few-shot sample in langchain to generate the prompt instruction corresponding to step 4.4.

[0065] A specific embodiment of the prompt instruction is as follows: "You are an expert in the cigarette knowledge domain. Please extract the cigarette entities and relationships in the following text into a knowledge graph structure. The extracted graph triple structure is identified by (label:`**`{name:"@@"})-[label:`**`]-(label:`**`{name:"@@"}), and the extracted individual entities are identified by (label:`**`{name:"@@"}), where ** is the label of the entity or relationship you identified, and @@ is the name of the entity or relationship you identified."

[0066] [Fewshot Sample]

[0067] User: AAA is a new slim cigarette with flavor capsules, created by ZZZ as an "expert in blending tobacco and tea."

[0068] Step 6) Return the corresponding results through the large model knowledge relationship extraction module.

[0069] The extraction results include: (label:`Tobacco Product`{name:”AAA”})-[label:`Cigarette Circumference`]-(label:`Thickness`{name:”Slim”}); (label:`Tobacco Product`{name:”AAA”})-[label:`Whether it has a menthol capsule`]-(label:`Menthol capsule`{name:”Menthol capsule”})).

[0070] Step 7) Generate the corresponding Cypher statement based on Step 5 for the construction of the knowledge graph.

[0071] The Cypher statements mentioned include: $create(n:`Tobacco Product`{name:”AAA”})-[r:`Cigarette Circumference`]-(m:`Thickness`{name:”Slim”}); $create(n:`Tobacco Product`{name:”AAA”})-[r:`Whether it has a menthol capsule`]-(m:`Menthol capsule`{name:”Menthol capsule”}).

[0072] Step 8) Perform deduplication and disambiguation on the entity content involved in the Cypher statement.

[0073] The deduplication and disambiguation mentioned above refer to the following: In the tobacco industry, the names of cigarettes and some other fields are often colloquially used. For example, the cigarette brand "CCC (White General)" is commonly referred to as "White General" in the BBB region. In the preliminary data collection work, a cigarette terminology table and an alias table have been constructed. During the entity deduplication and disambiguation stage, the entity names involved in the cypher statements will be corrected for disambiguation. For example, if a user asks "What does White General taste like?", the large model in the cypher statement generation stage under the prompt will return the cypher statement "match(m:`Tobacco Products`{name:“White General”})-[r:`Taste`]-(n)return n;". The entity deduplication and disambiguation stage will correct the cypher statement result to obtain "match(m:`Tobacco Products`{name:“CCC White General”})-[r:`Taste`]-(n)return n;". The corrected result can correctly match the entity references in the knowledge graph, thus providing accurate results. In addition, for structures that already exist in the knowledge graph, a Cypher query will be used to perform pattern matching with the nodes to be generated to check for duplicates. For example, in this embodiment, the Cypher query statement for the nodes to be generated is "match(n:`Tobacco Product`{name:"AAA"})-[r:`Cigarette Circumference`]-(m:`Thick and Thin Cigarette`{name:"Slim Cigarette"})return count((n)-[r]->(m)); $match(n:`Tobacco Product`{name:"AAA"})-[r:`Whether it has a menthol capsule`]-(m:`Menthol capsule`{name:"Menthol capsule"})return count((n)-[r]->(m));".

[0074] Through specific experiments, the Meta-llama3-8B-Instruct, fine-tuned with LoRa, was used as the base model. The simulation environment included a GPU: Quadro RTX 6000, a maximum memory limit of 8GB, and CUDA version: CUDA compilationtools, release 12.1, V12.1.105. To test the accuracy of the knowledge graph structure construction method, a test set of 389 natural language texts in the tobacco field was prepared. The test set data sources included texts from various China Tobacco Corporation websites, tobacco evaluation texts from major tobacco websites such as Yanyue.com and Tobacco Market Network, and dialogue texts from offline tobacco retail stores. The test set was input into both the general model and this invention for testing, and the extracted knowledge graph structure was manually evaluated. The precision, recall, and F1 score of the evaluation results were calculated, and the results are shown in Table 2.

[0075] Table 2: Method Precision Recall F1 General-GPT-Task 0.77152 0.59736 0.67336 This method 0.93881 0.89536 0.91669

[0076] Compared with existing technologies, in terms of functionality, this invention can achieve automated graph structure extraction and knowledge graph structure construction. In terms of performance, tested using a self-built test set, this invention achieves an accuracy of 93.8% in the graph structure extraction task, a 16.7% improvement over the general model, and an F1 score of 91.669, a 24.2% improvement over the general model.

[0077] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for constructing a cigarette knowledge base based on a large language model and knowledge graph, characterized in that, After collecting and labeling data, a pre-trained model is obtained by fine-tuning the large language model using LoRA. This model generates a graph structure based on prompt words constructed from natural language text, which is then used to generate corresponding Cypher statements and import them into the database.

2. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to claim 1, characterized in that, The training set data for LoRA fine-tuning is derived from cigarette-related text data collected in a multimodal manner.

3. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to claim 1, characterized in that, The prompt words are generated quickly using langchain's FewShotPromptTemplete, and the few-shot method is used to enhance the prompt words. The constructed prompt task includes task labeling, few-shot samples (annotated original text and expected output), and user input text.

4. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to claim 1, characterized in that, The pre-trained model constructs a task based on the knowledge graph structure, extracting the knowledge graph structure from the input text.

5. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to claim 1, characterized in that, The graph relation database mentioned is the neo4j graph database, which stores relevant knowledge graph data using a graph data model.

6. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to any one of claims 1-5, characterized in that, specifically... include: Step 1) Collect raw data through a multi-source modal approach, including: web crawling, recording of conversations with offline cigarette retail stores, and descriptions from official cigarette documentation; Step 2) Model common-sense conceptual knowledge graphs for knowledge in the tobacco retail scenario, specifically including: 2.1) To construct a cigarette knowledge base, establish a knowledge graph that covers relevant knowledge content in the cigarette retail scenario as comprehensively as possible; 2.2) Based on the above three dimensions, a top-down graph structure will be constructed; Step 3) The collected data is labeled with questions and answers, and the labeled data is used for LoRA fine-tuning to enhance the model's question-answering ability in the tobacco vertical field; The aforementioned question-and-answer annotation includes: annotating dialogue entities and dialogue content; Step 4) Perform formatting operations before model extraction; The formatting process described above involves formatting the user's natural language text to facilitate subsequent regular expression matching. Step 5) Construct large language model prompt words using the FewShotPromptTemplete component from the langchain component library to realize the transformation from natural language to knowledge graph structure, specifically including: 5.1) Perform graph structure annotation on natural language text to serve as the expected response of the large language model in the few-shot samples; 5.2) Define the prompt task in the prefix task; 5.3) Combine the labeled data with the text segments to form a few-shot sample; 5.4) Read the natural language text content corresponding to the user input; 5.5) Call the template of the few-shot sample in langchain to generate the prompt instruction corresponding to step 4.4; Step 6) Return the corresponding results through the large model knowledge relationship extraction module; Step 7) Generate the corresponding Cypher statement based on Step 5 for knowledge graph construction; Step 8) Perform deduplication and disambiguation on the entity content involved in the Cypher statement.

7. The method for constructing a cigarette knowledge base based on a large language model and knowledge graph according to claim 7, characterized in that, The graph structure annotation includes: for entity nodes in the graph, annotation in the form of (label:`**`{name:”@@”}), where ** is the entity label to be extracted and @@ is the entity node to be extracted; for relationships in the graph, annotation in the form of [label:`**`], where ** is the relationship label to be extracted, and the relationship category is set by experts in the cigarette industry based on the cigarette specification dimension.

8. A cigarette knowledge base construction system based on a large language model and knowledge graph, implementing the method of any one of claims 1-7, characterized in that, include: The system comprises an instruction construction and generation module, a large-scale model knowledge relation extraction module, a cypher statement generation module, and a graph construction and disambiguation module. Specifically: the instruction construction and generation module generates corresponding instructions based on the natural language text input from the system, inputs the constructed instructions into the large-scale model, and the trained model returns the extracted knowledge triples for the text and the corresponding graph structure. The cypher statement generation module further processes the graph structure returned by the model, constructing corresponding cypher statements from the graph structure and passing the cypher statements to the graph construction and disambiguation module. The graph construction and disambiguation module performs disambiguation processing on the entity content designed for the cypher statement in the current graph structure. If there is no cypher statement triple structure to be inserted in the current graph structure, the current knowledge triple structure is inserted into the knowledge graph.

Images