An LLM problem optimization method, medium and system combined with enterprise portraits

Abstract

The present invention provides an LLM question optimization method, medium and system combined with enterprise portraits, belonging to the technical field of enterprise portraits, including: First, the system constructs an enterprise knowledge graph containing enterprise basic information, business fields, product services, etc. Then, the system extracts keywords in the user's question and matches relevant nodes and relationships in the knowledge graph to form preliminary enterprise portrait elements. After that, the system uses the HyperLogLog algorithm and the Word2Vec model to evaluate the relevance between the keywords and the enterprise portrait elements and perform semantic expansion to obtain more extensive enterprise portrait elements. Finally, the system integrates the question text and the enterprise portrait elements, generates an optimized prompt, inputs it into the language model to obtain a preliminary answer, and post-processes the answer based on the enterprise portrait elements to output the final optimized result, solving the technical problem that the LLM answer deviates from the expected answer of the questioner.

Description

Technical Field

[0001] The present invention belongs to the technical field of enterprise portraits. Specifically, it relates to an LLM problem optimization method, medium, and system combined with enterprise portraits. Background Art

[0002] In recent years, large language models (LLMs) have been widely used in various application scenarios, such as question - answering systems, digital assistants, content generation, etc., and have become one of the important technologies in the field of artificial intelligence. LLMs obtain powerful language understanding and generation capabilities by learning from large - scale corpora and can provide users with high - quality natural language interaction services. However, there are also some problems in the actual application of LLMs.

[0003] First of all, most LLMs are trained based on general corpora and lack in - depth understanding of specific domains or enterprises, making it impossible to fully understand the targeted questions raised by users. The answers given by LLMs may be too general to meet the specific needs of users. This situation is particularly prominent in vertical industry applications, such as the financial and medical fields, where the questions raised by users often involve professional knowledge and industry logic, and it is difficult for LLMs to give accurate and valuable answers.

[0004] Secondly, the content output by LLMs may have problems such as factual errors, logical confusion, or inappropriate expressions. This is because although LLMs perform well in language generation, they lack in - depth understanding of real - world knowledge. In some application scenarios, these defects may lead to serious consequences, such as giving wrong advice in medical health consultations. Therefore, it is necessary to conduct in - depth content analysis and correction of the output of LLMs.

[0005] Thirdly, LLMs lack pertinence when answering questions and often cannot well capture the key points of the questions and the real needs of users. If the background and context of the questions cannot be accurately understood, the answers given by LLMs may not be practical.

[0006] In summary, in the prior art, due to the fact that it is often difficult for questioners to ask highly targeted questions, there is a technical problem that the answers of LLMs deviate from the expected answers of questioners. Summary of the Invention

[0007] In view of this, the present invention provides an LLM problem optimization method, medium, and system combined with enterprise portraits, which can solve the technical problem in the prior art that due to the fact that it is often difficult for questioners to ask highly targeted questions, the answers of LLMs deviate from the expected answers of questioners.

[0008] The present invention is implemented as follows:

[0009] The first aspect of the present invention provides an LLM question optimization method combined with an enterprise portrait, including the following steps:

[0010] S10. Construct an enterprise knowledge graph, including enterprise basic information, business fields, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners;

[0011] S20. Obtain the question text of the questioner and extract the keyword set in the question text;

[0012] S30. For each keyword in the keyword set, perform a match in the enterprise knowledge graph, and take the nodes and relationships in the enterprise knowledge graph with a matching degree greater than the first threshold as enterprise portrait elements, denoted as the first elements;

[0013] S40. Use the HyperLogLog algorithm to perform cardinality estimation on the first elements to quickly evaluate the relevance of each keyword to the first elements;

[0014] S50. Utilize the Word2Vec model, combined with the simulated annealing algorithm, use the keywords with a relevance greater than the second threshold as seeds, and perform semantic expansion on the seeds to obtain a set of semantically related expanded keywords;

[0015] S60. Based on the set of expanded keywords, perform a secondary match in the enterprise knowledge graph, and take the nodes and relationships in the enterprise knowledge graph with a matching degree greater than the third threshold as enterprise portrait elements, denoted as the second elements;

[0016] S70. Use a pre-trained prompt generation model, input the question text and the second elements for fusion, and generate an optimized prompt;

[0017] S80. Input the optimized prompt into the LLM model to obtain a preliminary answer, and post-process the preliminary answer based on the second elements, including entity alignment, fact checking, and content supplementation, and finally output an optimized answer result highly relevant to the enterprise portrait.

[0018] Specifically, the step S10 specifically includes: collecting relevant data on the basic information, business fields, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners of the enterprise, and modeling this information into a knowledge graph. The knowledge graph consists of nodes (representing entities) and edges (representing the relationships between entities). Various types of information of the enterprise are abstracted into different types of nodes, and the relationships between the nodes are also modeled, so as to integrate various types of information of the enterprise into a structured knowledge base, laying a foundation for subsequent question optimization.

[0019] Among them, the specific steps of step S20 include: First, obtain the original question text from the questioner, and then apply natural language processing techniques to perform preprocessing operations such as word segmentation and part-of-speech tagging on the question text, so as to extract the keyword set in the question text. Keywords usually refer to the words in the question that can best reflect the question theme and information needs. The methods for extracting keywords include word frequency statistics, text structure features, semantic similarity, etc. Through this step, a keyword set for subsequent matching and expansion is obtained.

[0020] Among them, the specific steps of step S30 include: Traverse each keyword in the keyword set and perform a matching search in the previously constructed enterprise knowledge graph. For each keyword, calculate its matching degree in the knowledge graph. If the matching degree is greater than the first threshold (the reference value is 0.75), then record the knowledge graph nodes and relationships corresponding to this keyword as the first elements. Through this step, enterprise knowledge graph elements that match the question keywords are obtained, laying a foundation for subsequent semantic expansion and answer optimization.

[0021] Among them, the specific steps of step S40 include: Use the HyperLogLog algorithm to estimate the cardinality of the first elements (nodes and relationships) extracted from the knowledge graph. The HyperLogLog algorithm is a probabilistic algorithm for estimating the cardinality (number of different elements) of a data set, and its result reflects the correlation degree of each keyword with the first elements: The larger the cardinality, the more graph elements the keyword is related to. In this way, the correlation degree of each keyword with the enterprise portrait can be quickly evaluated, providing a basis for subsequent semantic expansion.

[0022] Among them, the specific steps of step S50 include: First, use the pre-trained Word2Vec model to obtain the semantic Embedding vectors of each keyword, and then use the simulated annealing algorithm to optimize these Embedding vectors. Use the keywords with a correlation degree greater than the second threshold (the reference value is 0.65) as seeds, and search for semantically related extended keywords through the simulated annealing algorithm. The obtained extended keyword set not only contains the keywords of the original question, but also contains semantically related vocabulary, providing richer clues for subsequent secondary matching.

[0023] Among them, the specific steps of step S60 include: Traverse the extended keyword set obtained from the previous step and perform a second matching search in the enterprise knowledge graph. For each extended keyword, calculate its matching degree in the knowledge graph. If the matching degree is greater than the third threshold (the reference value is 0.85), then record the knowledge graph nodes and relationships corresponding to this extended keyword as the second elements. Through this step, a set of enterprise portrait elements that are more semantically related to the question are obtained, providing more accurate input for subsequent answer optimization.

[0024] Among them, the specific steps of the step S70 include: using a pre-trained prompt generation model, taking the original question text and the second element (enterprise portrait elements related in semantics) obtained from the previous step as inputs, and generating an optimized prompt. This optimized prompt can better reflect the intention of the question and be combined with the enterprise portrait information, providing a basis for the subsequent optimization of LLM question answering.

[0025] Among them, the specific steps of the step S80 include: inputting the optimized prompt generated in the step S70 into a pre-trained large language model (LLM) to generate a preliminary answer result. Then, use the previously obtained second element (enterprise portrait related in semantics) to post-process the preliminary answer result, including entity alignment, fact checking, and content supplementation. Through this series of post-processing steps, a high-quality optimized answer result highly relevant to the enterprise portrait is finally output.

[0026] On the basis of the above technical solution, an LLM question optimization method combining enterprise portrait of the present invention can also be improved as follows:

[0027] Among them, the obtaining steps of the first threshold include:

[0028] Collect a large number of enterprise knowledge graph matching sample data;

[0029] Manually annotate the matching degree scores (between 0 and 1) in the samples;

[0030] Use machine learning algorithms (such as decision trees, random forests, etc.) to train the samples and establish a matching degree prediction model;

[0031] Test different thresholds on the validation set, and select the threshold with the highest F1 score as the first threshold.

[0032] The obtaining steps of the second threshold include:

[0033] Construct a keyword-enterprise element relevance evaluation data set;

[0034] Use the expert scoring method to quantitatively evaluate the relevance (0-100 points);

[0035] Adopt the ROC curve analysis method to calculate the true positive rate and false positive rate under different thresholds;

[0036] Select the threshold corresponding to the point with the largest Youden's Index as the second threshold.

[0037] The obtaining steps of the third threshold include:

[0038] Collect a large number of secondary matching samples of enterprise knowledge graphs;

[0039] Adopt five-fold cross-validation and divide the samples into a training set and a test set;

[0040] Use the grid search method on the training set to test the matching accuracy at different thresholds;

[0041] Select the threshold with the highest accuracy on the test set as the third threshold.

[0042] The range of the first threshold: 0.6 - 0.9, the optimal value: 0.75;

[0043] The second threshold: the range: 0.5 - 0.8, the optimal value: 0.65;

[0044] The third threshold: the range: 0.7 - 0.95, the optimal value: 0.85.

[0045] The enterprise knowledge graph is represented as:

[0046] , where V represents the set of nodes, and E represents the set of edges; the node v ∈ V represents various information entities of the enterprise, including at least products, business fields, and organizational structures; the edge e ∈ E represents the semantic relationship between entities, including at least belonging to and cooperation.

[0047] Among them, the method of keyword extraction adopts keyword extraction based on word frequency statistics:

[0048] , where, represents the keyword k i The word frequency of the keyword k in the question text Q of the questioner, represents the keyword k i The number of times it appears in Q, |Q| represents the total number of words in the question text, and sort the values and select the top n as the keyword set K, where the keyword set in the question text .

[0049] Among them, the calculation of the matching degree is expressed as:

[0050] ; where, represents the keyword k i The similarity between and the node v.

[0051] Among them, the HyperLogLog algorithm is used to estimate the cardinality of the first element. Specifically: the HyperLogLog algorithm first maps each element x i to a 32-bit random number through the hash function h , and then records The number ρ of consecutive 0s at the highest bit of ; Finally, by averaging the ρ values to estimate the cardinality |S|: ; where m is the number of buckets, is a correction constant.

[0052] Among them, the objective function of the simulated annealing algorithm is defined as:

[0053] ; where represents the cosine similarity of the word vectors of keyword k i and k j ; is the word vector representation of keyword k i ; is the word vector representation of keyword k j ; represents is a d-dimensional real vector, and d represents the dimension of the word vector of k i ;

[0054] Furthermore, the string similarity uses the edit distance or cosine similarity; the semantic similarity utilizes the pre-trained word vector model.

[0055] Furthermore, the prompt generation model uses the generative language model of Transformer as the basic model.

[0056] The second aspect of the present invention provides a computer-readable storage medium, wherein program instructions are stored in the computer-readable storage medium, and when the program instructions run on a computer, they are used to execute the above-mentioned LLM problem optimization method combined with enterprise portraits.

[0057] The third aspect of the present invention provides an LLM problem optimization system combined with enterprise portraits, which includes the above-mentioned computer-readable storage medium.

[0058] Compared with the prior art, the beneficial effects of the LLM problem optimization method, medium and system combined with enterprise portraits provided by the present invention are as follows: It can make full use of the enterprise knowledge graph, deeply understand the problem background, and improve the pertinence of the answers generated by the LLM. At the same time, through multiple rounds of keyword extraction, semantic expansion and prompt optimization, it can more accurately capture the user's needs, reduce factual errors and logical confusion in the content output by the LLM. In addition, the content analysis and correction in the post-processing link further ensure the accuracy of the answer results, solving the technical problem in the prior art that due to the fact that it is often difficult for the questioner to pose a very targeted question, there is a deviation between the LLM answer and the expected answer of the questioner. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of the method provided by the present invention. Detailed implementation manners

[0060] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0061] As Figure 1 shown, it is a flowchart of an LLM problem optimization method combining enterprise portraits provided by the present invention. This method includes the following steps:

[0062] S10. Construct an enterprise knowledge graph, including enterprise basic information, business fields, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners;

[0063] S20. Obtain the question text of the questioner and extract the keyword set in the question text;

[0064] S30. For each keyword in the keyword set, perform matching in the enterprise knowledge graph, and use the nodes and relationships in the enterprise knowledge graph with a matching degree greater than the first threshold as enterprise portrait elements, denoted as the first elements;

[0065] S40. Use the HyperLogLog algorithm to estimate the cardinality of the first elements and quickly evaluate the relevance of each keyword to the first elements;

[0066] S50. Use the Word2Vec model and combine the simulated annealing algorithm. Using the keywords with a relevance greater than the second threshold as seeds, perform semantic expansion on the seeds to obtain a set of semantically related expanded keywords;

[0067] S60. Based on the set of expanded keywords, perform secondary matching in the enterprise knowledge graph, and use the nodes and relationships in the enterprise knowledge graph with a matching degree greater than the third threshold as enterprise portrait elements, denoted as the second elements;

[0068] S70. Use a pre-trained prompt generation model, input the question text and the second elements for fusion, and generate an optimized prompt;

[0069] S80. Input the optimized prompt into the LLM model to obtain a preliminary answer, and post-process the preliminary answer based on the second elements, including entity alignment, fact checking, and content supplementation, and finally output an optimized answer result highly relevant to the enterprise portrait.

[0070] The following will describe in detail the specific implementation manners of the above steps:

[0071] The specific implementation of step S10 is as follows: Construct an enterprise knowledge graph. First, collect relevant data such as the basic information, business areas, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners of the enterprise. Then, model this information into a knowledge graph. A knowledge graph is a structured data representation form, consisting of nodes (representing entities) and edges (representing the relationships between entities). When constructing an enterprise knowledge graph, various types of information of the enterprise will be abstracted into different types of nodes, and the relationships between the nodes will also be modeled. For example, there can be a "belongs to" relationship between the "product service" node and the "business area" node. In this way, various types of information of the enterprise are integrated into a structured knowledge base, laying a foundation for subsequent problem optimization.

[0072] The specific implementation of step S20 is as follows: Obtain the question text of the questioner and extract the keyword set from the question text. First, obtain the original question text from the questioner. Then, apply natural language processing techniques to perform preprocessing operations such as word segmentation and part-of-speech tagging on the question text, so as to extract the keyword set from the question text. Keywords usually refer to the words in the question that can best reflect the question theme and information needs. The methods for extracting keywords include: keyword extraction based on word frequency statistics, keyword extraction based on text structure features, keyword expansion based on semantic similarity, etc. Through this step, a keyword set for subsequent matching and expansion is obtained.

[0073] The specific implementation of step S30 is as follows: For each keyword in the keyword set, perform a match in the enterprise knowledge graph, and take the nodes and relationships in the enterprise knowledge graph with a matching degree greater than the first threshold as enterprise portrait elements, denoted as the first elements. First, traverse each keyword in the keyword set. For each keyword, perform a matching search in the previously constructed enterprise knowledge graph. The matching methods can be based on string similarity, semantic similarity, etc. For each keyword, calculate its matching degree in the knowledge graph. If the matching degree is greater than the first threshold (the reference value is 0.75), then record the knowledge graph nodes and relationships corresponding to this keyword as the first elements. Through this step, enterprise knowledge graph elements matching the question keywords are obtained, laying a foundation for subsequent semantic expansion and answer optimization.

[0074] The specific implementation of step S40 is as follows: The HyperLogLog algorithm is used to estimate the cardinality of the first element, and quickly evaluate the relevance of each keyword to the first element. The HyperLogLog algorithm is a probabilistic algorithm for estimating the cardinality (number of distinct elements) of a data set. In this step, the HyperLogLog algorithm is used to estimate the cardinality of the first element (nodes and relationships) extracted from the knowledge graph. The cardinality estimation result reflects the degree of relevance of each keyword to the first element: the larger the cardinality, the more graph elements the keyword is related to. In this way, the relevance of each keyword to the enterprise portrait can be quickly evaluated, providing a basis for subsequent semantic expansion. The advantage of the HyperLogLog algorithm is its low space complexity and fast calculation speed, which is very suitable for processing large-scale graph data.

[0075] The specific implementation of step S50 is as follows: Using the Word2Vec model, combined with the simulated annealing algorithm, taking the keywords with a relevance greater than the second threshold as seeds, semantic expansion of the seeds is performed to obtain a set of semantically related expanded keywords. First, the pre-trained Word2Vec model is used to obtain the semantic Embedding vector of each keyword. Word2Vec is a neural network-based word embedding model that can capture the semantic similarity between words. Then, the simulated annealing algorithm is used to optimize these Embedding vectors. The simulated annealing algorithm is a heuristic optimization algorithm that explores the solution space by simulating the annealing process to find the optimal solution. In this step, the keywords with a relevance greater than the second threshold (reference value is 0.65) are used as seeds, and semantically related expanded keywords are searched through the simulated annealing algorithm. The obtained set of expanded keywords not only contains the keywords of the original problem, but also contains semantically related vocabulary, providing richer clues for subsequent secondary matching.

[0076] The specific implementation of step S60 is as follows: Based on the set of expanded keywords, secondary matching is performed in the enterprise knowledge graph, and the nodes and relationships of the enterprise knowledge graph with a matching degree greater than the third threshold are used as enterprise portrait elements, denoted as the second element. First, traverse the set of expanded keywords obtained from the previous step. For each expanded keyword, a second matching search is performed in the enterprise knowledge graph. The purpose of this matching is to find graph elements that are closer to the expanded keyword. For each expanded keyword, calculate its matching degree in the knowledge graph. If the matching degree is greater than the third threshold (reference value is 0.85), then the knowledge graph nodes and relationships corresponding to the expanded keyword are recorded as the second element. Through this step, a set of enterprise portrait elements that are more semantically related to the problem is obtained, providing more accurate input for subsequent answer optimization.

[0077] The specific implementation of step S70 is as follows: Using a pre-trained prompt generation model, the problem text and the second element are input and fused to generate an optimized prompt. First, use a pre-trained prompt generation model. The prompt generation model is a technology based on the generative language model, which can generate an optimized prompt according to the input text. In this step, the original problem text and the second element (semantically related enterprise portrait elements) obtained from the previous step are used as inputs and input into the prompt generation model. The prompt generation model will generate an optimized prompt based on these inputs. This optimized prompt can better reflect the intention of the problem and be combined with the enterprise portrait information. Through this step, a high-quality prompt is obtained, providing a basis for the subsequent LLM question and answer optimization.

[0078] The specific implementation of step S80 is as follows: The optimized prompt is input into the LLM model to obtain a preliminary answer, and the preliminary answer is post-processed based on the second element, including entity alignment, fact checking, and content supplementation, and finally an optimized answer result highly relevant to the enterprise portrait is output. First, the optimized prompt generated in step S70 is input into a pre-trained large language model (LLM). The LLM model will generate a preliminary answer result according to the prompt. Then, the second element (semantically related enterprise portrait) obtained previously is used to post-process the preliminary answer result. This includes: 1) Entity alignment, aligning the entities in the answer with the entities in the knowledge graph to ensure that the information in the answer is consistent with the enterprise portrait; 2) Fact checking, using the information in the knowledge graph to check the facts of the answer and correct possible errors; 3) Content supplementation, identifying the missing information in the answer and supplementing it with the knowledge graph. Through this series of post-processing steps, a high-quality optimized answer result highly relevant to the enterprise portrait is finally output.

[0079] Generally speaking, this LLM question optimization method combined with the enterprise portrait makes full use of a number of cutting-edge technologies such as knowledge graph technology, semantic expansion algorithm, and prompt generation to achieve precise optimization of the LLM output results. Among them, the first threshold is obtained by training a machine learning model, the second threshold is obtained by using the ROC curve analysis method, and the third threshold is obtained by using the grid search method. These methods provide a scientific and reliable basis for the determination of the threshold. The entire optimization process closely revolves around the enterprise portrait to ensure that the LLM output can highly match the actual situation and needs of the enterprise, providing high-quality question and answer services for the enterprise.

[0080] To better understand and implement the present invention, a specific embodiment 1 of the method in the present invention is provided below. The specific implementation manners of this embodiment 1 are described as follows: The specific implementation manner of step S10 is as follows:

[0081] First, it is necessary to construct an enterprise knowledge graph, which includes relevant data such as the basic information, business fields, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners of the enterprise. The knowledge graph is a structured data representation form, consisting of nodes (representing entities) and edges (representing the relationships between entities).

[0082] Let the enterprise knowledge graph be represented as , where V represents the set of nodes and E represents the set of edges; the node v ∈ V represents various information entities of the enterprise, including at least products, business fields, and organizational structure; the edge e ∈ E represents the semantic relationships between entities, including at least belonging to and cooperation.

[0083] Graph database technologies such as Neo4j and OrientDB can be used to store and manage this enterprise knowledge graph. Through appropriate data modeling and relationship definition, various information of the enterprise is integrated into a structured knowledge base, laying a foundation for subsequent problem optimization.

[0084] The specific implementation manner of step S20 is as follows:

[0085] First, it is necessary to obtain the question text Q of the questioner. Then, natural language processing technology is applied to perform preprocessing operations such as word segmentation and part-of-speech tagging on the question text, so as to extract the keyword set in the question text .

[0086] The methods for keyword extraction can be as follows:

[0087] 1. Keyword extraction based on word frequency statistics:

[0088] ,

[0089] where represents the word frequency of the keyword k i in the question text Q, represents the number of times the keyword k i appears in Q, |Q| represents the total number of words in the question text, and values are sorted and the top n are selected as the keyword set K.

[0090] 2. Keyword extraction based on text structure features:

[0091] Train a keyword classification model using features such as the position of words in a sentence, the key nature of a title, and the word frequency in a sentence, and classify the words in the question text into keywords and non - keywords.

[0092] 3. Keyword expansion based on semantic similarity:

[0093] Adopt a pre - trained word vector model, such as Word2Vec, GloVe, etc., to calculate the semantic similarity of words in the question text. Incorporate words with semantic similarity to the original keywords into the keyword set K.

[0094] Through the above methods, the keyword set K for subsequent matching and expansion is finally obtained.

[0095] The specific implementation of step S30 is as follows:

[0096] For each keyword k in the keyword set i , it is necessary to perform a match in the enterprise knowledge graph and calculate its matching degree . If the matching degree is greater than the first threshold θ1, record the knowledge graph nodes and relationships corresponding to the keyword as the first element set F1.

[0097] The matching degree can be calculated using the following formula:

[0098] ;

[0099] where represents the similarity between the keyword k i and the node v. The similarity can be calculated based on methods such as string similarity and semantic similarity.

[0100] String similarity can use metrics such as edit distance and cosine similarity:

[0101] ;

[0102] or

[0103] ;

[0104] Semantic similarity can utilize a pre - trained word vector model, such as Word2Vec:

[0105] ;

[0106] where and respectively represent the word vector representations of the keyword k i and the node v.

[0107] Through this step, the enterprise knowledge graph element set F1 matching the problem keywords is obtained, laying a foundation for subsequent semantic expansion and answer optimization. The determination of the first threshold θ1 can be obtained through methods such as machine learning model training, and the reference value is 0.75.

[0108] The specific implementation of step S40 is as follows:

[0109] To quickly evaluate the relevance of each keyword to the first element set F1, the HyperLogLog algorithm is used to estimate the cardinality of F1. The HyperLogLog algorithm is a probabilistic algorithm for estimating the cardinality (number of different elements) of a data set, and its core idea is as follows:

[0110] Let the original data set be , and it is desired to estimate its cardinality . The HyperLogLog algorithm first maps each element x i to a 32-bit random number through a hash function h , and then records the number of consecutive 0s in the highest bit of ; finally, the cardinality is estimated through the average of the ρ values : ;

[0111] where m is the number of buckets (usually 64 or 128), is a correction constant.

[0112] In this step, the HyperLogLog algorithm is applied to the first element set F1 for cardinality estimation, and the cardinality i corresponding to each keyword k is obtained. This cardinality value reflects the degree of relevance of the keyword k i to the first element: the larger the cardinality, the more graph elements the keyword is related to.

[0113] Through this method, the relevance of each keyword to the enterprise portrait can be quickly evaluated, providing a basis for subsequent semantic expansion. The advantage of the HyperLogLog algorithm is its low space complexity and fast calculation speed, which is very suitable for processing large-scale graph data.

[0114] The specific implementation of step S50 is as follows:

[0115] In this step, using the Word2Vec model and the simulated annealing algorithm, keywords with a relevance greater than the second threshold θ2 are used as seeds, and these seed keywords are semantically expanded to obtain a set K2 of semantically related expanded keywords.

[0116] First, use the pre-trained Word2Vec model to obtain the semantic Embedding vector of each keyword k i . . Word2Vec is a neural network-based word embedding model that can capture semantic similarities between words. Its core idea is to learn the low-dimensional semantic representation vectors of each word by maximizing the prediction probability of the word's context.

[0117] Then, use the simulated annealing algorithm to optimize these Embedding vectors. The simulated annealing algorithm is a heuristic optimization algorithm that explores the solution space by simulating the metal annealing process to find the optimal solution. In this step, keywords with a relevance greater than the second threshold θ2 (reference value 0.65) are used as seeds, and the simulated annealing algorithm is used to search for semantically related extended keywords. The objective function of the simulated annealing algorithm can be defined as:

[0118] ;

[0119] where, represents the cosine similarity of the word vectors of keyword k i and k j .

[0120] In this way, a set of semantically related extended keywords K2 is obtained, which not only contains the keywords of the original problem but also contains semantically related vocabulary, providing richer clues for the subsequent secondary matching.

[0121] The specific implementation of step S60 is as follows:

[0122] Based on the set of extended keywords K2 obtained from the previous step, a second matching needs to be performed in the enterprise knowledge graph , and the matching degree of each extended keyword k i with the graph is calculated . If the matching degree is greater than the third threshold θ3, the knowledge graph nodes and relationships corresponding to the extended keyword are recorded as the second element set F2.

[0123] The calculation formula of the matching degree is the same as that in step S30:

[0124] ;

[0125] where, represents the similarity between keyword k i and node v, which can be calculated based on methods such as string similarity or semantic similarity.

[0126] Through this step, a set of enterprise portrait element sets F2 that are more semantically relevant to the question is obtained, providing a more accurate input for subsequent answer optimization. The determination of the third threshold θ3 can be obtained through methods such as grid search, and the reference value is 0.85.

[0127] The specific implementation of step S70 is as follows:

[0128] In this step, a pre-trained prompt generation model is used to take the original question text Q and the second element set F2 obtained from the previous step as inputs, and generate an optimized prompt P.

[0129] The prompt generation model is a technology based on the generative language model, which can generate an optimized prompt according to the input text. Its core idea is to train a conditional generation model , learning how to generate a high-quality prompt P according to the question text Q and the enterprise portrait element F2. This optimized prompt P will better reflect the intention of the question and be combined with the enterprise portrait information.

[0130] A generative language model based on Transformer, or iFlytek Spark, ERNIE Bot, etc. can be used as the basis of the prompt generation model. During training, a large number of question-answer pair samples are collected, where Q is the question text, P is the corresponding high-quality prompt, and A is the correct answer. The model parameters are trained by maximizing the likelihood function :

[0131] ;

[0132] where θ represents the model parameters and N is the number of training samples.

[0133] The trained prompt generation model can, in actual applications, use the question text Q and the enterprise portrait element F2 as inputs to generate an optimized prompt P, providing a basis for subsequent LLM question-answer optimization.

[0134] The specific implementation of step S80 is as follows:

[0135] Finally, the optimized prompt P generated in step S70 is input into a pre-trained large language model (LLM) to obtain a preliminary answer result A0. Then, the second element set F2 (semantically relevant enterprise portrait) obtained previously is used to post-process A0, including entity alignment, fact checking, and content supplementation, and finally an optimized answer result A that is highly relevant to the enterprise portrait is output.

[0136] Specifically, the post - processing includes the following steps:

[0137] 1. Entity alignment: Identify the entities in answer A0 and align them with the entities in the knowledge graph G to ensure that the information in the answer is consistent with the enterprise portrait. Rule - based or learning - based entity alignment methods can be used.

[0138] 2. Fact checking: Utilize the information in the knowledge graph G to check the facts in answer A0, discover possible errors or inconsistencies, and make corrections. Question - answering techniques based on knowledge bases can be adopted.

[0139] 3. Content supplementation: Identify the missing information in answer A0 and supplement it with the content of the knowledge graph G to make the final output answer A more complete and rich. Content supplementation methods based on generative language models can be used.

[0140] Through this series of post - processing steps, a high - quality optimized answer result A that is highly relevant to the enterprise portrait is finally output to meet the user's information needs.

[0141] The second aspect of the present invention provides a computer - readable storage medium, in which program instructions are stored. When the program instructions run on a computer, they are used to execute the above - mentioned LLM question optimization method combined with an enterprise portrait.

[0142] The third aspect of the present invention provides an LLM question optimization system combined with an enterprise portrait, which includes the above - mentioned computer - readable storage medium.

[0143] Specifically, the principle of the present invention is as follows:

[0144] First of all, the enterprise knowledge graph can provide a more professional and practical knowledge base for the question - answering system. Traditional LLM models are trained based on general corpora and lack professional domain knowledge for specific industries or enterprises. The enterprise knowledge graph constructed in the present invention encompasses various important data such as the basic information, business situation, and product services of the enterprise, and can truly reflect the actual situation of the enterprise. When analyzing questions and generating answers, by using this professional knowledge graph, it is possible to better understand the semantics of questions and find answers that are more in line with the actual situation of the enterprise.

[0145] Secondly, the method of multi-round keyword matching and semantic expansion can capture the information needs of the question more comprehensively. Relying solely on the keywords in the question text may not fully understand the true intention of the user. The present invention extracts keywords in multiple ways based on word frequency statistics, text structure features, semantic similarity, etc., and semantically expands the keywords through the simulated annealing algorithm to obtain a richer keyword set. These keywords not only include the words in the original question but also cover other semantically related concepts, providing more clues for subsequent enterprise portrait matching.

[0146] Thirdly, the application of the prompt generation model enables the output of the LLM system to better fit the actual situation of the enterprise. The prompt is the input of the LLM model and directly affects the quality of its output result. The present invention adopts a pre-trained prompt generation model to generate an optimized prompt according to the question text and enterprise portrait information. This prompt can better express the true intention of the question and combine with the enterprise's own information to provide high-quality input for the LLM model, thereby generating a more practical reply.

[0147] Finally, the post-processing step of the answer based on the enterprise knowledge graph further ensures the accuracy and integrity of the output result. The LLM model itself has certain biases and errors, and the preliminary reply generated may have factual problems or information gaps. The present invention uses the enterprise knowledge graph to perform entity alignment, fact checking, and content supplementation on the reply result, eliminating these problems and making the finally output answer fully fit the actual situation of the enterprise.

[0148] Generally speaking, the LLM question optimization method proposed by the present invention skillfully combines multiple cutting-edge technologies such as enterprise knowledge graph, natural language processing, and prompt generation, effectively improving the performance of the question-and-answer system in terms of pertinence, reliability, and integrity.

[0149] To better understand and implement the present invention, the following provides Example 2 of a specific application scenario of the present invention: In Example 2, the company name is "XX Intelligence", which is mainly engaged in the research and development and sales of industrial automation equipment and control systems. As a leading domestic provider of industrial Internet solutions, XX Intelligence has rich industry experience and a professional technical team, providing efficient production management and equipment maintenance services for numerous manufacturing enterprises.

[0150] 1. Construct an enterprise knowledge graph

[0151] First, it is necessary to construct an enterprise knowledge graph for XX Intelligence. By collecting relevant data such as the company's basic information, business areas, product services, organizational structure, core technologies, market positioning, competitive advantages, development strategies, and partners, an enterprise knowledge graph as shown in Table 1 was established.

[0152] Table 1 XX Intelligence Enterprise Knowledge Graph

[0153]

[0154] This knowledge graph covers various important information of XX Intelligence Company, laying a solid foundation for subsequent problem analysis and answer optimization.

[0155] 2. User Question Analysis and Keyword Extraction

[0156] A manufacturing customer, Xiaoming, encountered a problem when using the equipment remote diagnosis system of XX Intelligence and submitted the following consultation to the company:

[0157] "The servo motors on our company's production line frequently malfunction, resulting in a significant decline in production efficiency. Does XX Intelligence have any targeted solutions? I want to solve this problem as soon as possible and improve production stability."

[0158] For the problem raised by Xiaoming, it is first necessary to analyze it and extract keywords. Through natural language processing technology, the following keyword set is extracted from the problem text:

[0159]

[0160] These keywords cover the main content of Xiaoming's consultation, including key information such as fault symptoms, impact indicators, and demand requests. Next, it is necessary to use these keywords to match and expand in the enterprise knowledge graph to obtain more abundant enterprise portrait information, laying a foundation for subsequent answer optimization.

[0161] 3. First-round Keyword Matching and Cardinality Evaluation

[0162] For each keyword in the keyword set K, a matching search is performed in the previously constructed XX Intelligence enterprise knowledge graph. The specific matching results are shown in Table 2:

[0163] Table 2 First-round Keyword Matching Results

[0164]

[0165] As can be seen from Table 2, each keyword has found corresponding nodes and relationships in the knowledge graph, and the matching degree is above 0.7, indicating that these keywords are closely related to the business and technology of XX Intelligence. Record these matching results as the first element set , providing a basis for subsequent semantic expansion.

[0166] Next, the HyperLogLog algorithm is used to estimate the cardinality of the first element set to quickly evaluate the relevance of each keyword to the enterprise profile. The calculation results are shown in Table 3:

[0167] Table 3 Keyword-Enterprise Profile Relevance Evaluation

[0168]

[0169] As can be seen from Table 3, the estimated cardinality value of the keyword "solution" is the highest, indicating that it is most closely related to various elements involved in the enterprise profile. While the cardinality of "production efficiency" is relatively low, indicating that its association with the enterprise profile is relatively weak. These results provide a basis for subsequent semantic expansion.

[0170] 4. Semantic Expansion and Second Round of Matching

[0171] Based on the keyword-enterprise profile relevance evaluation results obtained in the previous step, keywords with relatively high relevance (estimated cardinality value greater than 30) are selected as seeds, and semantic expansion is performed using the Word2Vec model and the simulated annealing algorithm to obtain the expanded keyword set shown in Table 4 :

[0172] Table 4 Expanded Keyword Set

[0173]

[0174] These expanded keywords not only include the keywords in the original question but also involve broader semantic concepts, providing richer clues for the subsequent second-round matching.

[0175] Next, in the enterprise knowledge graph of XX Intelligence, the expanded keyword set is retrieved for matching again. The specific results are shown in Table 5:

[0176] Table 5 Second Round of Keyword Matching Results

[0177]

[0178] As can be seen from Table 5, the matching degree of this round of matching results is generally higher, involving richer enterprise profile elements. Record these matching results as the second element set , providing a basis for subsequent prompt generation and answer optimization.

[0179] 5. Prompt Generation and Answer Optimization

[0180] With the aforementioned keyword matching and semantic expansion results, a pre-trained prompt generation model can be utilized to generate an optimized prompt, which is provided as input to the LLM question-answering system.

[0181] Specifically, the original question text Q of Xiaoming and the second element set are input together into the prompt generation model, and the training objective is to maximize the likelihood function . After training, the prompt generation model outputs the following optimized prompt P:

[0182] "XX Intelligent Company has rich industrial automation equipment and solutions, including products such as industrial robots, intelligent production lines, and remote diagnostic systems. Regarding the problems frequently reported by customers about servo motor failures and declining production efficiency, what specific solutions does XX Intelligent have? What mature technologies and services does the company have in aspects such as motor fault diagnosis, production line optimization, and equipment preventive maintenance? It is hoped that it can help customers quickly improve production stability and achieve digital transformation."

[0183] This optimized prompt not only contains the key information of the original question but also fully integrates the professional advantages of XX Intelligent in aspects such as industrial automation, remote diagnosis, and production optimization, providing a good input basis for the LLM model to generate high-quality answers.

[0184] Finally, this optimized prompt P is input into the pre-trained LLM model to obtain a preliminary answer result . Then, using the previously constructed enterprise knowledge graph, it is further optimized and improved:

[0185] 1. Entity alignment: Identify various entities mentioned in the answer, such as "servo motor", "production line", "remote diagnostic system", etc., and align them with the corresponding nodes in the knowledge graph to ensure the accuracy of information.

[0186] 2. Fact checking: Use the detailed information in the knowledge graph to conduct factual verification on various technologies, services, etc. involved in the answer, and discover and correct possible errors or incompleteness.

[0187] 3. Content supplementation: Analyze the information that may be missing in the answer, such as the specific solutions of the company in aspects such as motor fault diagnosis and production line optimization, and supplement it using the knowledge graph.

[0188] After these post-processing steps, a high-quality optimized answer A is finally obtained, and the main content is as follows:

[0189] "Thank you very much for consulting XX Intelligence about relevant solutions. Regarding the problems of frequent servo motor failures and decreased production efficiency on your company's production line, the following specific solutions are available:

[0190] 1. Motor Fault Diagnosis and Preventive Maintenance

[0191] XX Intelligence has developed a remote diagnosis system based on industrial Internet of Things technology, which can monitor the operation status of equipment in real time and use big data analysis to predict possible failures. The service team also provides equipment fault analysis and preventive maintenance solutions to help customers minimize equipment failure rates and improve production line reliability.

[0192] 2. Intelligent Production Line Optimization

[0193] XX Intelligence's intelligent production line solution integrates core technologies such as machine vision and industrial control, enabling the intelligentization and automation of the production process. Through the optimized configuration and process monitoring of the production line, equipment utilization and production efficiency can be significantly improved. At the same time, XX Intelligence's factory operation management system can provide customers with comprehensive data analysis and production line optimization suggestions.

[0194] 3. Digital Transformation Services

[0195] XX Intelligence has been deeply involved in the industrial automation field for many years, accumulating rich industry experience and technical advantages. It can provide you with customized industrial Internet solutions, including functions such as equipment remote monitoring, fault warning, and production optimization, helping enterprises achieve digital transformation and improve overall production stability and competitiveness.

[0196] I hope the above solutions can help you solve the current production problems as soon as possible. If you have any other needs, please feel free to contact us."

[0197] Through this optimized answer A, not only detailed solutions are given to the specific problems raised by Xiaoming, but also the core technical advantages and service capabilities of XX Intelligence in the industrial automation field are fully demonstrated. This LLM question optimization method based on the enterprise knowledge graph ensures the professionalism, integrity, and pertinence of the answer content, providing high-quality information services to customers.

[0198] The above is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A LLM problem optimization method combined with enterprise portrait, characterized in that: The following steps are involved: S10. Build an enterprise knowledge graph, including basic enterprise information, business areas, products and services, organizational structure, core technologies, market positioning, competitive advantages, development strategies and partners; S20, obtaining the question text of the questioner, and extracting a set of keywords in the question text; S30, for each keyword in the keyword set, matching is performed in the enterprise knowledge graph, and nodes and relationships in the enterprise knowledge graph with a matching degree greater than a first threshold are taken as enterprise profile elements, recorded as first elements; S40, using the HyperLogLog algorithm to perform cardinality estimation on the first factor, and quickly evaluating the relevance of each keyword to the first factor; S50, using the Word2Vec model in combination with a simulated annealing algorithm, taking keywords with a relevance greater than a second threshold as seeds, performing semantic expansion on the seeds to obtain a semantically related set of expanded keywords; S60, performing secondary matching in the enterprise knowledge graph based on the extended keyword set, taking the nodes and relationships of the enterprise knowledge graph whose matching degree is greater than a third threshold as enterprise portrait elements, recorded as second elements; S70, using a pre-trained prompt generation model, inputting the question text and the second element for fusion to generate an optimized prompt; S80, inputting the optimized prompt into the LLM model to obtain a preliminary answer, and post-processing the preliminary answer based on the second factor, including entity alignment, fact checking, and content supplementation, and finally outputting an optimized answer result that is highly relevant to the enterprise portrait; The enterprise knowledge graph is represented as: G = (V, E), where V represents a node set and E represents an edge set; a node v∈V represents various information entities of an enterprise, including at least products, business areas, and organizational structures; an edge e∈E represents a semantic relationship between entities, including at least "belongs to" and "cooperation"; Among them, the keyword extraction method adopts keyword extraction based on word frequency statistics: Among them, TF(k i ) indicates keyword k i The frequency of words in the questioner's question text Q, f(k i ,Q) represents keyword k i The number of times it appears in Q, |Q| represents the total number of words in the question text, and TF(k i ) values ​​and select the first n as the keyword set K, where the keyword set K in the question text = {k1, k2, ..., k n }; The calculation of matching degree is expressed as: Among them, similarity(k i ,v) indicates keyword k i The similarity with node v; Among them, string similarity uses edit distance or cosine similarity; semantic similarity uses pre-trained word vector model; The HyperLogLog algorithm is used to estimate the cardinality of the first element. Specifically, the HyperLogLog algorithm first converts each element x i Mapped to a 32-bit random number h(x i ), then record h(x i ) is the number of consecutive 0s in the highest digit, ρ, that is, ρ(h(x i )); Finally, by averaging the ρ values To estimate the cardinality Where m is the number of buckets, α m is the correction constant; Among them, the objective function of the simulated annealing algorithm is defined as: in, Indicates keyword k i and k j The cosine similarity of word vectors; For keyword k i The word vector representation of For keyword k j The word vector representation of ; express is a d-dimensional real vector, where d represents k i The dimension of the word vector; The prompt generation model uses the Transformer generative language model as the basic model.

2. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are run in a computer, they are used to execute the LLM problem optimization method combined with enterprise portraits as described in claim 1.

3. An LLM problem optimization system combined with enterprise portrait, characterized in that: The computer-readable storage medium of claim 2 is included.

Images