A product configuration knowledge question-answering method based on the fusion of knowledge graph and large language model

Abstract

This invention discloses a product configuration knowledge question-answering method based on the fusion of knowledge graph and large language model, belonging to the field of knowledge question-answering technology. The method includes: obtaining intent entities and constructing a final intent entity set; selecting nodes related to the user's question from the product configuration knowledge graph, constructing an initial knowledge graph node set, building an initial knowledge graph subgraph, and pruning it based on node attributes to obtain a final knowledge graph node subgraph; finally, using a large language model to classify and answer questions based on information in the final knowledge graph node subgraph. This invention combines knowledge graph, large language model, deep learning, and prompting learning methods to design a question-answering framework for the product configuration domain, which has shown good results in product configuration question-answering experiments.

Description

Technical Field

[0001] This invention relates to the field of knowledge question answering technology, and also to the field of artificial intelligence technology, particularly to knowledge graphs and large language models, specifically a product configuration knowledge question answering method based on the fusion of knowledge graphs and large language models. Background Technology

[0002] In the field of product configuration, providing customers with configuration solutions tailored to their needs is one of the core goals for enterprises to enhance market competitiveness and achieve sustainable development. However, the product configuration knowledge accumulated within enterprises is often scattered across different documents or systems, exhibiting significant fragmentation. This fragmentation makes the retrieval and access of relevant knowledge reliant on extensive manual operations, not only consuming considerable time costs but also directly leading to low product configuration efficiency and difficulty in quickly responding to customer needs. Therefore, how to build a system capable of efficiently acquiring configuration knowledge and achieving intelligent question answering has become a key direction for promoting the intelligentization of product configuration.

[0003] With the rapid development of technology, large language models (LLMs) have been widely applied across various industries. However, the inherent limitations of insufficient domain knowledge, architectural flaws, and the randomness of reasoning in LLMs can lead to deceptive outputs, or "illusions." Furthermore, LLMs are black-box models, often unable to capture and acquire factual knowledge. To improve the adaptability of LLMs in domain-specific question-answering tasks, retrieval augmentation (RAG) methods have been proposed. These methods enhance the model's knowledge coverage by introducing external knowledge bases for retrieval before reasoning. This approach not only avoids the high cost of retraining the model for each task but also alleviates the "illusion" problem of LLMs to some extent, improving the accuracy and relevance of answers. However, traditional retrieval augmentation (RAG) methods are mostly based on unstructured text retrieval, failing to fully consider the relationships between knowledge points, thus limiting their effectiveness in complex question-answering scenarios. In contrast, knowledge graphs explicitly represent entities and their relationships in a structured form, possessing good interpretability and reasoning capabilities. Combining knowledge graphs with large language models can not only compensate for the shortcomings of large language models in terms of factual and logical consistency, but also enhance their ability to understand domain-specific issues through structured knowledge. The addition of knowledge graphs not only improves the granularity of information matching, but also provides large language models with verifiable and traceable external knowledge support, thereby achieving higher-quality and more interpretable question-and-answer output.

[0004] Existing knowledge-based question-answering methods, such as those based on knowledge graphs and large language models, typically use knowledge graphs as a source of structured knowledge and large language models for question understanding and answer generation, thus improving the intelligence level of question-answering systems to some extent. However, these methods are mostly geared towards general knowledge-based question-answering scenarios and lack systematic modeling for product configuration question-answering tasks. Furthermore, existing solutions generally do not differentiate between configuration question-answering questions of varying complexity, employing a uniform generation method for simple query-type questions and those requiring multi-step reasoning, making it difficult to balance response efficiency and result reliability. Moreover, their research focus is primarily on the knowledge retrieval and answer generation stages, lacking targeted design for the identification process of domain intent and entities in user questions. In existing solutions, intent and entity recognition are usually treated as auxiliary steps in the question-answering process, often relying on rule-based methods, single deep learning models, or directly performed by large language models. They fail to integrate large language models with sequence labeling-based deep learning models in the intent and entity recognition stage, making it difficult to fully leverage the complementary advantages of different models. This results in insufficient accuracy and stability of recognition results when facing user questions with diverse entity types and flexible language expressions in the product configuration domain. Summary of the Invention

[0005] To address at least one of the aforementioned problems, this invention provides a product configuration knowledge question-answering method based on the fusion of knowledge graphs and large language models.

[0006] To achieve the above objectives, the specific solution of the present invention is as follows: A product configuration knowledge question answering method based on the fusion of knowledge graph and large language model includes the following steps: S1. The user's question intent is identified by using a large language model and a pre-trained BERT-BiLSTM-Attention-CRF model respectively. Then, the recognition results of the two are concatenated and fed into the large language model for intent entity filtering to obtain the final intent entity and build the final intent entity set. S2. Filter out nodes related to user questions from the product configuration knowledge graph, build an initial knowledge graph node set, and construct an initial knowledge graph subgraph. Each node in the initial knowledge graph subgraph includes its corresponding relationship and attributes. Check whether all intent entities in the final intent entity set are nodes in the product configuration knowledge graph. If so, prune the initial knowledge graph subgraph. If not, keep it as is and obtain the optimized final knowledge graph node subgraph. S3. Based on the number of triples in the final knowledge graph subgraph, user questions are classified into three types: invalid queries, simple queries, and multi-hop reasoning. For invalid queries, the model refuses to answer. For simple queries, the big language model directly answers based on the node information in the knowledge graph subgraph. For multi-hop reasoning, the big language model learns hints from the nodes, relationships, and attributes in the knowledge graph subgraph, and combines the CoT (Cooperation of Knowledge) mechanism to guide the big language model to complete knowledge reasoning and generate answers.

[0007] In one specific embodiment of the present invention, step S2 involves filtering nodes related to user questions from the product configuration knowledge graph, including the following operations: S21. Obtain the intent entity embedding vector by processing user questions through a pre-trained BERT model; S22. Based on the KG-BERT model, the structured information of the product configuration knowledge graph is transformed into knowledge graph node embedding vectors; S23. Based on the similarity between the intent entity embedding vector and the embedding vectors of each knowledge graph node, select the knowledge graph node embedding vectors with high similarity to form the initial knowledge graph node set corresponding to the user question.

[0008] In one specific embodiment of the present invention, in step S23, the similarity between the intention entity embedding vector and the embedding vectors of each knowledge graph node is measured by cosine similarity, the calculation formula of which is as follows: In the formula, Embedded vectors for intent entities With the n Embedded vectors of knowledge graph nodes The cosine similarity.

[0009] In one specific embodiment of the present invention, step S2, the pruning operation includes the following steps: B21. Input the final intent entity set, the initial knowledge graph node set, the attribute set and associated relation set of each node in the initial knowledge graph node set, and the pruning similarity threshold for nodes, relations, and attributes. Initialize the set of reserved nodes, the set of reserved relationships, and the set of reserved attributes; mark the intent entities in the final intent entity set that consist entirely of letters and numbers as code-sensitive entities; B22. Using each node in the initial knowledge graph node set as a candidate, calculate the semantic similarity between each candidate and each intent entity. Add the candidate with a similarity greater than the node pruning similarity threshold to the retained node set to update the retained node set. B23. Using each relation of each node in the current set of retained nodes as a candidate quantity, calculate the semantic similarity between each candidate quantity and each intent entity. Add the knowledge graph triples corresponding to the candidate quantities with similarity greater than the relation pruning similarity threshold to the set of retained relations to update the set of retained relations. B24. Structural Association Completion: For a relationship in the set of retained relationships, if only one end node is located in the set of retained nodes, and the other end is a code-sensitive entity located in the initial knowledge graph node set, then the other end node is added to the set of retained nodes to ensure path integrity. B25. Using each attribute of each node in the current set of retained nodes as a candidate, calculate the semantic similarity between each candidate and each intent entity, and add the candidate with a similarity greater than the attribute pruning similarity threshold to the set of retained relations to update the set of retained relations. S26. Output the current set of retained nodes, the set of retained relations, and the set of retained attributes as the final knowledge graph subgraph.

[0010] As a specific embodiment of the present invention, the formula for calculating the semantic similarity of concept matching is as follows: In the formula, According to and Edit distance is used to calculate semantic similarity between concepts. Represented as and The edit distance between the two concepts , They represent and The lengths of the two concept strings.

[0011] In a specific embodiment of the present invention, in step S3, for simple queries, the large language model is used to directly answer based on the nodes in the knowledge graph subgraph. Specifically, this includes: directly injecting the nodes, their relationships, and attributes in the final knowledge graph subgraph into the prompt template of the large language model, and the large language model directly organizing the relationships or attributes between the nodes and providing the answer.

[0012] As a specific implementation of the present invention, for multi-hop reasoning, the large language model learns hints from three aspects: nodes, relationships, and attributes in the knowledge graph subgraph. Combined with the CoT (Coding over Time) mechanism, it guides the large language model to complete knowledge reasoning and generate answers. Specifically, this includes: first, based on the node types in the knowledge graph configured for the product, constructing standard hint templates for each type of node; then, inputting the node information and node types from the knowledge graph subgraph into the large language model, which selects the corresponding hint template and fills in the node information to generate a reasoning thought chain; finally, guided by this thought chain, the large language model generates answers in conjunction with the knowledge graph subgraph.

[0013] Beneficial Effects: This invention proposes a product configuration question-answering framework combining knowledge graphs and large language models. The framework specifically includes the following three design aspects: 1) Using a structured product configuration knowledge graph as the foundation for the knowledge question-answering system, improving the professionalism and accuracy of question answering; 2) Decomposing the product configuration knowledge question-answering task into three key steps: user intent recognition, semantic matching, and answer generation. In the answer generation process, considering the characteristics of questions in the product configuration domain, questions are divided into simple query questions and multi-level reasoning questions, and corresponding answer generation strategies are designed for each type of question; 3) Introducing a large language model to assist in completing intent entity recognition and answer generation for multi-level reasoning questions. Specifically, in the user intent recognition stage, a fusion entity recognition method is proposed, combining BERT-BiLSTM-Attention-CRF and a large language model. In the answer generation stage of multi-level reasoning questions, a prompting learning and thought chain mechanism is introduced to improve the response capability of the large language model in complex reasoning scenarios. Overall, the invention combines knowledge graphs, large language models, deep learning, and prompting learning to design a question-answering framework for the product configuration domain, which has shown good results in product configuration question-answering experiments. Attached Figure Description

[0014] Figure 1 This is a framework diagram of the product configuration knowledge question answering method based on the fusion of knowledge graph and large language model of the present invention; Figure 2 This is an example diagram of the intent entity filtering based on a large language model according to the present invention; Figure 3 This is a structural diagram of the BERT model of this invention; Figure 4 This is a comparison chart of the entity-level recognition evaluation results of four large language models in the case of this invention; Figure 5 This is a comparison chart of the comprehensive evaluation results of the four large language models in the case of this invention; Figure 6 This is a diagram showing the final result of intent entity recognition evaluation in the case of this invention. Figure 7 This is a comparison chart of the evaluation results of answer generation for four large language models in the case of this invention; Figure 8 This is a radar chart showing the answer generation and evaluation of four large language models in the case of this invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0016] A product configuration knowledge question answering method based on the fusion of knowledge graph and large language model, such as Figure 1 As shown, it includes the following steps: S1. The user's question intent is identified by using a large language model and a pre-trained BERT-BiLSTM-Attention-CRF model respectively. Then, the recognition results of the two are concatenated and fed into the large language model for intent entity filtering to obtain the final intent entity and build the final intent entity set. In this step, firstly, the BERT-BiLSTM-Attention-CRF model is pre-trained using a historical product configuration dataset to better recognize user intent. Then, the pre-trained BERT-BiLSTM-Attention-CRF model and a large language model are used respectively to perform preliminary intent entity recognition on the user's question. The intent entities obtained through these two methods (some may overlap) are concatenated. It is then necessary to determine which intent entities have a high probability of being the true intent entities. Therefore, from the intent entity sets obtained by these two methods, the large language model is then used to determine which intent entities have a high probability of being the true intent entities. Specifically, as follows... Figure 2 As shown, the list of candidate intent entities is filled into the prompt template, and the large language model completes the task of selecting high-probability intent entities from the candidate intent entities. Its output is the final set of intent entities, that is, all intent entities. In the formula, For user issues The set of intent entities obtained through a pre-trained BERT-BiLSTM-Attention-CRF model; For user issues The first [unclear] obtained through the BERT-BiLSTM-Attention-CRF model An entity with intent; For user issues The total number of intent entities obtained through the BERT-BiLSTM-Attention-CRF model; For user issues The set of intent entities obtained through a large language model; For user issues The first obtained through the large language model An entity with intent; For user issues The total number of intent entities obtained through the large language model; This refers to the set of end-user intent entities obtained through a large language model. The final result obtained by judging the two through a large language model is the first... A truly intentional entity, This represents the final total number of intent entities. S2. Filter out nodes related to user questions from the product configuration knowledge graph, build an initial knowledge graph node set, and construct an initial knowledge graph subgraph. Each node in the initial knowledge graph subgraph includes its corresponding relationship and attributes. Check whether all intent entities in the final intent entity set are nodes in the product configuration knowledge graph. If so, prune the initial knowledge graph subgraph. If not, keep it as is and obtain the optimized final knowledge graph node subgraph. This step involves "selecting nodes related to user questions from the product configuration knowledge graph", which includes the following operations: processing user questions through a pre-trained BERT model to obtain intent entity embedding vectors; converting the structured information of the product configuration knowledge graph into knowledge graph node embedding vectors based on the KG-BERT model; and selecting knowledge graph node embedding vectors with high similarity as the knowledge graph node vectors corresponding to user questions based on the similarity between the intent entity embedding vectors and the embedding vectors of each knowledge graph node, thus forming the corresponding initial knowledge graph node set. Specifically: In this step, BERT, as a pre-trained language model, utilizes a bidirectional Transformer structure and self-attention mechanism to effectively capture contextual dependencies, thereby generating deep semantic representations. The BERT model here is trained using a historical product configuration dataset to generate embedding vectors for intent entities, thus improving the accuracy of configuration recommendations and semantic understanding. During the pre-training phase, BERT learns word-level and sentence-level semantic information through the Masked Language Model and Next SentencePrediction tasks, respectively. First, the user question is segmented, with [CLS] and [SEP] inserted before and after. The pre-processed complete user question (including entity context) is then used as input, and TokenEmbeddings, Segment Embeddings, and Position Embeddings are performed to generate text representation vectors. The structure of the BERT model is as follows: Figure 3 As shown, a self-attention mechanism is used to calculate the association weights of each token with all other tokens, and a token vector containing contextual information is generated by weighted summation. The output of each layer is fed into the next layer after residual connection and layer normalization, finally obtaining a token vector matrix containing deep contextual semantics. Finally, the average vector of the tokens corresponding to the final intent entity is extracted from the output matrix as the embedding result of the intent entity.

[0017] In this step, KG-BERT is a variant of BERT optimized for knowledge graph embedding tasks. Its core is to transform the structured information of the knowledge graph into semantic vectors through a pre-training fine-tuning mechanism. Here, the knowledge graph is a pre-built personalized product configuration knowledge graph, containing two main categories of nodes: customer needs and configuration schemes. The customer needs category nodes include nodes for need source, need category, need name, and need item type. The configuration scheme category nodes include nodes for product, module, module instance, and quality characteristic type. Let the set of all node types be denoted as […]. ,in Indicates the first Each node type This represents the total number of node types. Different node types are connected through predefined relationships, as shown in Table 1. The model first converts the triple information (including head node, relationship, and tail node) in the knowledge graph into a text sequence that BERT can understand. This specifically includes the concatenation of two types of structured text: <node, relationship, node> and <node, attribute, attribute value>. 1) For text of the type <node, relation, node>, the attribute-attribute value is the intrinsic feature of the node. Therefore, during the concatenation process, the attributes and attribute values ​​of the node are superimposed and placed after the node information, and the parts are separated by [SEP]. For example, the node "engine M11" has the attribute "range" with the attribute value "100km". It has a "component" relationship with the node "configuration scheme 1". The node "configuration scheme 1" has the attribute "cost" with the attribute value "100,000". Then, the concatenation is as follows: "[CLS] Engine M11 [SEP] Range: 100km [SEP] Component [SEP] Configuration Scheme 1 [SEP] Cost: 100,000 [SEP]".

[0018] 2) For concatenating text of the type <node, attribute, attribute value>, use [SEP] to separate each part. For example, the node "engine M11" has the attribute "range" with the value "100Km", and is concatenated into the text "[CLS]engine M11[SEP]range[SEP]100Km[SEP]".

[0019] Similar to BERT, KG-BERT first performs Token Embeddings, Segment Embeddings, and Position Embeddings on the input text. Based on a pre-trained BERT model, KG-BERT is fine-tuned using knowledge graph triples to enhance relational semantics, making the model more focused on the connections between nodes, relationships, and attributes. For node embeddings, the average of the token vectors corresponding to the node name is taken; for attribute and attribute value embeddings, the attribute value vectors can be extracted separately.

[0020] Table 1. Definition of Elements in the Knowledge Graph for Personalized Product Configuration In this step, the similarity between the intent entity embedding vector and the embedding vectors of each knowledge graph node is measured using cosine similarity, with a threshold set. k To determine if the similarity meets the requirements, embedding vectors of knowledge graph nodes with a similarity not lower than a threshold are selected as knowledge graph node vectors for the user's question. All knowledge graph node vectors that meet the conditions are combined to form an initial knowledge graph node set. If the similarity of all nodes is not greater than or equal to... k This means that the desired answer is not found in the knowledge graph.

[0021] The formula for calculating cosine similarity is as follows: In the formula, Embedded vectors for intent entities With the n Embedded vectors of knowledge graph nodes Cosine similarity; , represents the set of embedding vectors for knowledge graph nodes. The total number of nodes; In this step, if all the final intent entities obtained in step S1 are nodes in the product configuration knowledge graph, then the initial knowledge graph subgraph (composed of nodes, their corresponding relationships, and attributes) is pruned; otherwise, no pruning is performed. Specifically, if all the final intent entities corresponding to the user question (obtained in step 1) are nodes in the product configuration knowledge graph (i.e., the intent entities are of the type in the starting or ending nodes in Table 1), then the corresponding initial knowledge graph node set is node-driven; otherwise, it is attribute-driven. In the node-driven class, the question content radiates outward from the node as the center, along with relationships and attributes. However, in this process, due to the large amount of node information obtained, not all information is valid, so pruning is required. Attribute-driven pruning refers to finding relevant nodes from attributes when no nodes appear in the intent entities extracted from the question; this process does not require pruning.

[0022] Pruning includes the following steps: B1. Input the final intent entity set The initial knowledge graph node set obtained from the initial matching Each node attribute set Each node vector set of relationships Pruning similarity thresholds for nodes, relationships, and attributes ; B2. Initialize the set of reserved nodes Preservation of relation sets and reserved attribute set ; B3. Set up the final intent entity Entities containing only letters and numbers are marked as code-sensitive entities; B4. For each candidate node Calculate its relationship with each intent entity Concept matching semantic similarity The calculation formula is as follows: In the formula, According to and Edit distance is used to calculate semantic similarity between concepts. Represented as and The edit distance between the two concepts , They represent and The lengths of the two concept strings; B5. If Then the candidate nodes Add to the set of reserved nodes ; B6. Determine the set of nodes to retain. Check if it is an empty set. If it is, stop pruning (i.e., end pruning); otherwise, proceed to the next step. B7. For each reserved node : B7.1 Traverse all its relations Calculate its relationship with each intention entity Concept matching semantic similarity ; B7.2 If Then Corresponding triples Add to the set of retention relations Among them NT For the connected tail entity; B7.3 Structural Association Completion Mechanism: For already retained relations (the set of retained relations) If only one end node is matched (located in the reserved node set), and the other end is a code-sensitive entity and is in the initial knowledge graph node set obtained from the initial matching, then the other end node is added to the reserved node set to ensure path integrity. B7.4 Traverse all attributes of the reserved node Calculate this attribute With each intention entity Concept matching semantic similarity ; B7.5 If Then Add to the set of reserved attributes ; B8. Output the pruned knowledge graph subgraph: The set of nodes to be retained Relation sets and attribute set ; S3. Based on the number of triples involved in the knowledge graph subgraph obtained in step S2, the user questions are classified into three types: invalid queries, simple queries, and multi-hop reasoning. For invalid queries, the model directly refuses to answer and provides a reason for the refusal. For simple queries, the big language model directly answers based on the node information in the knowledge graph subgraph. For multi-hop reasoning, the big language model learns from the nodes, relationships, and attributes in the knowledge graph subgraph and combines the CoT (Cooperation of Knowledge) mechanism to guide the big language model to complete knowledge reasoning and generate answers. The specific classification criteria for user questions can be set as needed. For example, an invalid query can be defined as having zero triples. In this case, if the initial knowledge graph node set is empty or the node set retained after pruning is empty in step S2, it can be directly identified as an invalid query. Another example is that a simple query can be defined as having one triple, and multi-hop inference can be defined as having two or more triples.

[0023] Regarding the answer generation part, the question The set of node information obtained through semantic matching and pruning is ,in Indicate the problem The corresponding number t 1 node n’ This represents the total number of nodes. Each node... The valid attribute information contained within, along with its associated valid nodes, provides rich contextual support for the subsequent CoT inference process. Therefore, the nodes... ,relation and attributes The three key elements that constitute the construction of node information are the pruned subgraph. .

[0024] For simple query questions, the large language model generates the answer simply by injecting the nodes, relationships, and attributes of the obtained knowledge graph subgraph (including the nodes, their relationships, and attributes) into the prompt template. The prompt template can be represented as: a) {head node} Name} and {Tail Node} Name} is {relation} }relation; b) {Node {attributes} of name} Name} is {attribute} value}.

[0025] For multi-hop reasoning problems, a fine-tuning method based on prompt learning is adopted based on a large language model. Without adding additional parameters, the large model learns from three aspects: nodes, relationships, and attributes. Combined with the CoT mechanism of the thinking chain, the large language model is guided to complete knowledge reasoning and generate answers.

[0026] In the process of automatically generating CoT hints, it is crucial to select and construct the corresponding hint template based on the node types in the predefined Table 1. This process first starts from the CoT hint template preset library. The CoT hint template accurately selects templates that match the target node type and customizes designs for different node types (such as product type, module type, and module instance type). For example, when the target node type is a product type, the CoT hint template focuses on guiding the model to start from user needs and gradually analyze the functional goals that the product needs to meet and their corresponding module instance composition; when the node type is a module type, the template uses the module as a bridge between the product and the module instance, revealing the relationship between the two; and for the module instance type, the CoT hint template further guides the model to combine specific module instance parameters for querying or comparison, and to conduct step-by-step reasoning and feasibility judgment. Once a template is selected, the key content of the node information (including nodes, relationships, and their attributes) is filled into the template to achieve automatic generation of hint information. This process can be completed with the help of scripts or automated tools, ensuring a high degree of structural and semantic fit and task relevance of CoT hints, thereby significantly improving the efficiency and accuracy of large language models in handling complex reasoning tasks. The formal description of automatically constructing CoT hints is shown in the following calculation formula: In the formula, the mapping function For node type Return directly to the CoT prompt template y .Function Obtain node information entity And fill in the prompt template .

[0027] CoT's automatic build process includes the following steps: B1: Input knowledge graph subgraph Intent Entity Set Node information entity Node type Mapping functionality to CoT prompt templates All CoT prompt templates y The CoT prompt template preset library ; B2: Node type determination: Based on the knowledge graph subgraph obtained in step 2 Based on the intent entity in the user question, determine the main node types involved in the user question. Specifically, the intersection of nodes in the knowledge graph subgraph and intent entities in the user question is taken, and the node types mainly involved in the user question are determined based on the nodes in the intersection.

[0028] B3: Template Selection: From the CoT prompt template preset library Choose an appropriate CoT prompt template y Based on node type Use mapping function , ; B4: Template Filling: Integrating Node Information Entities Add specific elements (including nodes, relationships, and attributes) to the selected template. y middle; B5: Prompt Generation: After filling in the template, the generated CoT prompt will be displayed. It is used to guide large language models in reasoning and generating answers.

[0029] Guided by CoT, the large language model can gradually enhance the answer generation process, making up for its lack of professional knowledge, thereby achieving the effective generation of product configuration knowledge.

[0030] Below, we will use the product configuration of a certain automobile company as the experimental object. This company has established an internal personalized product configuration knowledge graph, containing 989 nodes and 15,870 relationships. This embodiment constructs a question-and-answer dataset based on this graph. The questions in the dataset cover basic vehicle configuration information, historical order information, etc., and the question-and-answer types include customer-demand-oriented solution recommendations, comparisons between configuration information, basic information retrieval, and data analysis based on configuration information. The constructed dataset contains a total of 3,603 entries, and some query examples are shown in Table 2. Assume the question set is... The set of correct answers for reference is ,but q =3603.

[0031] Table 2 Query Examples The detailed steps are as follows: Step 1: User Intent Recognition Experiment First, user intent recognition experiments were conducted using a dataset. The BERT-BiLSTM-Attention-CRF model was trained, and prompt learning and evaluation of large language models were performed based on the dataset. In this embodiment, four large language models were selected: ChatGLM3-6B, LLaMA2-7B, LLaMA3-8B, and Qwen2.5-7b. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. Experimental results are expressed as accuracy (…). P ), recall rate ( R )and F The calculation method for a single value is as follows: TP This indicates that the model recognizes a certain label, and that is indeed the label it identifies. FP This indicates that the model predicted a certain label, but the actual label was not that label; FN This indicates that the model predicted a label that was not actually a label, but the label was. TN This indicates that neither the model's prediction nor the actual situation is a certain label.

[0032] First, the BERT-BiLSTM-Attention-CRF named entity recognition model for product configuration was transferred to train a second time with intent entity extraction as the task. By setting different learning rates and batch sizes, the impact of different hyperparameter configurations on model performance was evaluated. The performance results of each model at the entity level under different hyperparameter settings were presented. The final selected model configuration is shown in Table 3. The accuracy, recall, and F1 score of this model were 82.69%, 84.65%, and 80.83%, respectively.

[0033] Table 3 BERT-BiLSTM-Attention-CRF Model Configuration The second step involves applying four large language models—ChatGLM3-6B, LLaMA2-7B, LLaMA3-8B, and Qwen2.5-7b—to the intent entity recognition task. The performance of these four models is evaluated using a test set, and the results are as follows: Figure 4 and Figure 5 As shown. Among the four large language models evaluated, Qwen2.5-7b performed best in terms of precision, recall, and F1 score for entity recognition, reaching 88.16%, 52.88%, and 66.11%, respectively. Although its response speed is slower than ChatGLM3... While Qwen2.5-7b outperforms the latter in entity recognition and token-level recognition accuracy, it is more suitable for Chinese product configuration intent entity recognition tasks. However, current large language models do not perform as well as BERT-BiLSTM-Attention-CRF in this task in terms of recall and F1 score. This may be because these large language models are generative models, and entity recognition is not their strong suit.

[0034] Finally, by filling the list of candidate intent entities into the prompt template, the Qwen2.5-7b model completes the task of selecting high-probability intent entities from the candidate intent entities. The evaluation results are as follows: Figure 6 As shown in Table 4, compared to BERT alone. BiLSTM Attention Compared to entity recognition results from CRF and standalone large language models, the two fusion strategies, while having lower recall rates than BERT, still achieved better results. BiLSTM Attention While using CRF, it achieved significant improvements in both accuracy and F1 score, making it the best overall performer. This suggests that BERT... BiLSTM Attention Combining CRF with large models can effectively enhance the ability of large language models to recognize intent entities, especially product configuration-related intent entities.

[0035] Table 4 Comparison of Evaluation Results of Three Types of Intent Entity Recognition Methods Step 2: Semantic Matching Experiment Semantic matching experiments were conducted based on the intent entity recognition results of the test set, with a threshold set. k The node threshold during pruning is 0.85. and attribute threshold The threshold is 0.85. Since the relationships in the knowledge graph used in this paper are all uniformly preset, while relationships in real-world problems are often not explicitly presented, appropriately lowering the threshold helps prevent some effective information from being removed during the pruning process. To ensure the quality of subsequent large-scale model answer generation, the evaluation of the retrieval content based on the semantic matching experiment results needs to consider both precision and recall, especially recall, as it is necessary to ensure that positive examples are selected as much as possible. Low precision leads to a large amount of redundant and interfering information in the retrieval results. This low-quality information not only fails to effectively enhance the knowledge of the large language model but may also interfere with its judgment, leading to incorrect output. On the other hand, low recall may miss key information, thus affecting the comprehensiveness and accuracy of the large language model's generated results. Therefore, this paper uses the matching precision and recall between the retrieved entity and entities in the standard query path as evaluation metrics, as shown in Table 5. Ultimately, the precision and recall of the semantic matching experiment in this embodiment were 0.7783 and 0.7308, respectively, indicating that this method can extract correct subgraphs to a certain extent and has a good pruning effect.

[0036] Table 5 Examples of semantic matching experiment evaluation Step 3: Answer Generation Experiment The subgraph generated by semantic matching is used as the context for the large language model's answer. The prompt in this example is as follows: "You are a product configuration knowledge graph question-and-answer assistant. Answer only based on the provided context; fabrication is prohibited. If the answer cannot be determined from the context, please output the following in the conclusion line: Answer: Sorry, I cannot answer this question. There is no relevant information in my knowledge base. The final conclusion must begin with 'Answer:' and state the conclusion completely in natural language (including the subject and key details), using fluent language that conforms to Chinese expression habits; if explanations or clarifications are needed, please provide them separately outside the conclusion. Values ​​should come first, followed by units, and units should be standardized; replace 'is' with '.'" Furthermore, if the subject and target attributes can be identified but the context is missing, output "Answer: Sorry, I cannot answer this question. There is no [attribute] information about [subject] in my knowledge base." For multi-hop question-answering scenarios, CoT is used for step-by-step reasoning guidance. It automatically builds large language model prompt templates for different node types. The CoT prompts for some node types are shown in the table below.

[0037] Table 6 Examples of CoT's automatic build suggestions To objectively evaluate the performance of large language models in the product configuration knowledge domain, ROUGE- was introduced in addition to precision, recall, and F1 score. N The ROUGE and BLEU metrics are used to evaluate the text similarity between the generated answer and the reference answer. N In the evaluation indicators N=1. ROUGE-1 measures the unigram similarity between the generated answer and the reference answer by calculating the degree of overlap, which measures the consistency of their main information, essentially the recall rate. BLEU, on the other hand, calculates how many of the generated answers contain the reference answer. -gram appears in the reference text to evaluate the overall fluency and accuracy of the generated text.

[0038] ROUGE- N The formula for calculating BLEU is as follows: In the formula, This represents the set of reference answers. For the generated candidate answers, For all candidate answers that have appeared in the previous answer, the length is... n Phrases, Among the candidate answers The number of times a phrase appears. BP is a length penalty factor used to prevent the generated text from being too short. For correction Accuracy, for The weight. Usually N’ =4, and Therefore, it is 1 / 4 here.

[0039] Table 7 summarizes the overall results of this answer generation experiment. Figure 7 A visual comparison of the various evaluation indicators was provided. Although Llama3 8b has a recall rate 2.38% lower than ChatGLM3-6B and its response speed is not the fastest among the four large language models, but its accuracy, F1 and BLEU scores are significantly higher, making it the best overall performer. Figure 8 The radar chart shows the average latency of four large language models after inverse normalization. ChatGLM3-6B's value is close to 1, indicating that it has the lowest average latency and the fastest response speed. Overall, answer generation is affected to some extent by the two preceding stages. Under these conditions, Llama3... 8b can still consistently produce relatively effective answers.

[0040] Table 7 Comparison of Response Generation Evaluation Results for Four Large Language Models The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included within the protection scope of the present invention.

Claims

1. A product configuration knowledge question and answer method based on knowledge graph and large language model fusion, characterized in that, Includes the following steps: S1. The user's question intent is identified by using a large language model and a pre-trained BERT-BiLSTM-Attention-CRF model respectively. Then, the recognition results of the two are concatenated and fed into the large language model for intent entity filtering to obtain the final intent entity and build the final intent entity set. S2. Filter out nodes related to user questions from the product configuration knowledge graph, form an initial knowledge graph node set, and construct an initial knowledge graph subgraph. Each node in the initial knowledge graph subgraph includes its corresponding relationship and attributes. Check whether all intent entities in the final intent entity set are nodes in the product configuration knowledge graph. If so, prune the initial knowledge graph subgraph. If not, keep it as is and obtain the optimized final knowledge graph node subgraph. The pruning process includes the following steps: B21、input the final intent entity set, the initial knowledge graph node set, the attribute set and the associated relationship set of each node in the initial knowledge graph node set, and the pruning similarity threshold of nodes, relationships and attributes ; Initialize the set of reserved nodes, the set of reserved relationships, and the set of reserved attributes; mark the intent entities in the final intent entity set that consist entirely of letters and numbers as code-sensitive entities; B22. Using each node in the initial knowledge graph node set as a candidate, calculate the semantic similarity between each candidate and each intent entity. Add the candidate with a similarity greater than the node pruning similarity threshold to the retained node set to update the retained node set. B23. Using each relation of each node in the current set of retained nodes as a candidate quantity, calculate the semantic similarity between each candidate quantity and each intent entity. Add the knowledge graph triples corresponding to the candidate quantities with similarity greater than the relation pruning similarity threshold to the set of retained relations to update the set of retained relations. B24. Structural Association Completion: For a relationship in the set of retained relationships, if only one end node is located in the set of retained nodes, and the other end is a code-sensitive entity located in the initial knowledge graph node set, then the other end node is added to the set of retained nodes to ensure path integrity. B25. Using each attribute of each node in the current set of retained nodes as a candidate, calculate the semantic similarity between each candidate and each intent entity, and add the candidate with a similarity greater than the attribute pruning similarity threshold to the set of retained relations to update the set of retained relations. S26. Output the current set of retained nodes, the set of retained relations, and the set of retained attributes as the final knowledge graph subgraph; S3. Classify user questions based on the number of triples in the final knowledge graph subgraphs into three types: invalid queries, simple queries, and multi-hop reasoning. The model refuses to respond to invalid queries. For simple queries, the large language model is used to directly answer based on the node information in the knowledge graph subgraph; For multi-hop reasoning, the large language model learns hints from three aspects: nodes, relationships, and attributes in the knowledge graph subgraph. Combined with the CoT (Coding On-the-Top) mechanism, it guides the large language model to complete knowledge reasoning and generate answers. Specifically, this involves: first, constructing standard hint templates for each type of node in the knowledge graph based on the product configuration; then, inputting the node information and node type from the knowledge graph subgraph into the large language model, which selects the corresponding hint template and fills in the node information to generate a reasoning thought chain; finally, guided by this thought chain, the large language model generates answers by combining the knowledge graph subgraph.

2. The product configuration knowledge question and answer method based on the fusion of a knowledge graph and a large language model according to claim 1, characterized in that, In step S2, nodes related to user questions are selected from the product configuration knowledge graph, including the following operations: S21. Obtain the intent entity embedding vector by processing user questions through a pre-trained BERT model; S22. Based on the KG-BERT model, the structured information of the product configuration knowledge graph is transformed into knowledge graph node embedding vectors; S23. Based on the similarity between the intent entity embedding vector and the embedding vectors of each knowledge graph node, select the knowledge graph node embedding vectors with high similarity to form the initial knowledge graph node set corresponding to the user question.

3. The product configuration knowledge question answering method based on the fusion of a knowledge graph and a large language model according to claim 2, characterized in that, In step S23, the similarity between the intent entity embedding vector and the embedding vectors of each knowledge graph node is measured by cosine similarity, which is calculated as follows: In the formula, is an intended entity embedding vector cosine similarity with the embedding vector of the n th knowledge graph node​ 4. The product configuration knowledge question-answering method based on the fusion of knowledge graph and large language model as described in claim 1, characterized in that, The formula for calculating semantic similarity in concept matching is as follows: In the formula, According to and Edit distance is used to calculate semantic similarity between concepts. Represented as and The edit distance between the two concepts , They represent and The lengths of the two concept strings.

5. The product configuration knowledge question answering method based on the fusion of a knowledge graph and a large language model according to claim 1, characterized in that, In step S3, for simple queries, the large language model is used to directly answer based on the nodes in the knowledge graph subgraph. Specifically, this includes: directly injecting the nodes, their relationships, and attributes in the final knowledge graph subgraph into the prompt template of the large language model, and the large language model directly organizing the relationships or attributes between the nodes and providing the answer.

Images