An auxiliary method for private domain knowledge entry in data analysis based on LLM

Abstract

The present invention provides a data analysis private domain knowledge entry auxiliary method based on LLM. For LLM in the field of data analysis, based on the proposed data analysis private domain knowledge entry auxiliary method, based on the general prompt word engineering skills (knowledge standard element framework) and the specific expression paradigm of domain knowledge, for each domain knowledge that is effective for data analysis generation tasks, it can directly infer the specific logic to be used when querying according to user questions or needs. It can help humans to complete the pre-entry of knowledge efficiently and with high quality. This knowledge will eventually be given to LLM as part of Prompt (prompt word) through the use of RAG, so the relevant knowledge content with clearer expression and more self-consistent logic can ensure that LLM can more accurately complete data analysis tasks such as SQL generation.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to an LLM-based data analysis private domain knowledge entry auxiliary method and an LLM-based data analysis private domain knowledge entry auxiliary device, electronic device and computer-readable storage medium. Background Art

[0002] During the development of LLM Agent (Large Language Model Agent), prompts are mainly built by professional prompt engineers. However, if RAG (Retrieval-Augmented Generation) is used at the same time, the quality of the knowledge cannot be controlled because engineers cannot help all users enter knowledge content. This knowledge will be referenced through retrieval as part of the prompt, which will greatly affect the performance of the final agent generation task.

[0003] In the field of data analysis, on the one hand, the correctness of the query logic described in the knowledge text strongly depends on the understanding and application of private domain related knowledge, including indicator caliber, data table definition, data table application method, etc.; on the other hand, the clarity of the text expression logic also requires expert experience in Prompt Engineering.

[0004] Therefore, the input of private domain knowledge for data analysis will be limited by the input work of the Prompt Engineer mentioned above, making it impossible to complete the construction of Prompt (prompt words) efficiently and with high quality, thus reducing the R&D and data analysis efficiency of LLM Agent. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides the following technical solutions:

[0006] On the one hand, a method for assisting data analysis private domain knowledge entry based on LLM is provided, the method being implemented by an electronic device, and the method comprising:

[0007] S1. Preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking several knowledge elements;

[0008] S2, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements;

[0009] S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression;

[0010] S4. The LLM Agent performs knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result.

[0011] Furthermore, the method for constructing the knowledge standard element framework includes:

[0012] Collect some data to analyze private domain knowledge;

[0013] Parsing the data to analyze private domain knowledge and obtain knowledge elements in different types of knowledge content;

[0014] Define the trigger words, query logic and applicable scope of each type of knowledge element, and bind them with the type of corresponding knowledge element to generate a data analysis private domain knowledge system;

[0015] The template design component is used to design and generate the knowledge standard element framework based on the data analysis private domain knowledge system.

[0016] Furthermore, S2, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements, including:

[0017] Obtain original knowledge text and pre-prepared data table information;

[0018] The LLM Agent identifies several knowledge elements in the original knowledge text based on the knowledge standard element framework, further extracts several keywords from the knowledge elements, and identifies keywords with missing fields;

[0019] Use a search engine to retrieve data knowledge related to the keyword with missing fields: data table name, field name and enumeration value;

[0020] The LLM Agent combines the data table information and the retrieved data knowledge to rewrite and complete the knowledge elements of the keywords with missing fields;

[0021] The rewritten and completed knowledge statement is output and sent to the user end, and the administrator confirms / corrects it, and obtains the knowledge statement with complete elements.

[0022] Furthermore, when rewriting and completing the knowledge elements of the keywords with missing fields, it also includes:

[0023] If there are knowledge elements that cannot be rewritten or completed, a marking symbol is called to mark the place where the knowledge elements that cannot be rewritten or completed are placed.

[0024] Further, S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression, including:

[0025] The LLM Agent identifies and extracts the knowledge elements and query logic in the original knowledge text based on the knowledge standard element framework;

[0026] Determine whether the original knowledge text contains the knowledge elements that are not core elements: if so, determine that the expressions are redundant and delete the knowledge elements that are not core elements; otherwise, abandon them;

[0027] Determine whether the query logic in the original knowledge text is the default logic under the common data analysis requirements in the subject; if so, optimize the expression of the query logic and put the default logic in front; otherwise, abandon it;

[0028] Output the clearly expressed knowledge text obtained after the above operation.

[0029] Further, S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression, and also includes:

[0030] The LLM Agent identifies and extracts the knowledge expression logic and various relational association words in the original knowledge text;

[0031] Based on the query logic in the code instance defined in the knowledge standard element framework, the knowledge expression logic in the original knowledge text is logically checked: if the logic is inconsistent, the knowledge expression logic in the original knowledge text is logically rewritten using the query logic in the code instance; otherwise, it is abandoned;

[0032] Based on the query logic of the association relationship defined in the knowledge standard element framework, rewriting the association words of various relationships in the original knowledge text;

[0033] Output the clearly expressed knowledge text obtained after the above operation.

[0034] Further, S4, the LLM Agent performs knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result, including:

[0035] Enter the clearly expressed knowledge text;

[0036] The LLM Agent searches the original knowledge base to see whether there are knowledge statements with the same or similar logic as the clearly expressed knowledge text:

[0037] If so, integrate the clearly expressed knowledge text into knowledge sentences with the same or similar logic;

[0038] Otherwise, it is marked as a knowledge statement to be optimized, and the LLM Agent is used to retrieve and mark the association relationship in the knowledge statement to be optimized. Based on the marked association relationship, the LLM Agent rewrites and integrates the knowledge statement to be optimized based on the retrieved knowledge statements with the same or similar logic; if there is a knowledge statement to be optimized whose association relationship is a logical conflict, the corresponding logical conflict marking result is output;

[0039] The knowledge integration result is output to the user end, and is confirmed / corrected by the administrator, and the original knowledge base is updated based on the feedback confirmation / correction result.

[0040] On the other hand, a data analysis private domain knowledge entry auxiliary device based on LLM is provided, and the data analysis private domain knowledge entry auxiliary device based on LLM is used to implement the data analysis private domain knowledge entry auxiliary method based on LLM, and the device includes:

[0041] An element template construction module is used to preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking a number of knowledge elements;

[0042] An element completion and clarification module is used for the LLM Agent to complete and clarify the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements;

[0043] The expression logic optimization module is used for the LLM Agent to rewrite and optimize the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression;

[0044] The knowledge integration and updating module is used for the LLM Agent to perform knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and update the original knowledge base based on the integration result.

[0045] On the other hand, an electronic device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned LLM-based data analysis private domain knowledge entry assistance methods is implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned LLM-based data analysis private domain knowledge entry assistance methods.

[0047] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0048] The present invention constructs a knowledge standard element framework by presetting and configures it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking several knowledge elements; the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain knowledge sentences with complete elements; the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression; the LLM Agent integrates the knowledge text with clear expression based on the knowledge sentence in the original knowledge base, and updates the original knowledge base based on the integration result. It can be used for LLM (Large Language Model) applications in the field of data analysis, based on the proposed data analysis private domain knowledge entry auxiliary method, based on the general prompt word engineering skills (knowledge standard element framework) and the specific expression paradigm of domain knowledge, for each domain knowledge that is effective for data analysis generation tasks, it can directly infer the specific logic to be used when querying according to user questions or needs. It can help humans to complete the pre-entry of knowledge efficiently and with high quality. This knowledge will eventually be given to LLM as part of the prompt through the use of RAG. Therefore, relevant knowledge content with clearer expression and more self-consistent logic can ensure that LLM can more accurately complete data analysis tasks such as SQL generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1It is a flow chart of a data analysis private domain knowledge entry auxiliary method based on LLM provided by an embodiment of the present invention;

[0051] Figure 2 It is a time sequence diagram of data analysis private domain knowledge entry based on LLM provided by an embodiment of the present invention;

[0052] Figure 3 It is a structural diagram of a knowledge standard element framework provided by an embodiment of the present invention;

[0053] Figure 4 It is a block diagram of a data analysis private domain knowledge entry auxiliary device based on LLM provided by an embodiment of the present invention;

[0054] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0056] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0057] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0058] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0059] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0060] ‌Definition and scope of private domain‌: The private domain refers to the resources owned by enterprises / institutions and other units. For example, the knowledge resources of an enterprise may involve the fields of business management, sales, marketing, operations, inventory, logistics, etc.

[0061] Importance of private domain knowledge management: The unique private domain knowledge system owned by an enterprise is one of its core competitive advantages, covering its long-term accumulated experience, skills, culture and other resources.

[0062] ‌Application of data analysis in private domain operations‌: Data analysis plays a key role in measuring the effectiveness of marketing activities and promoting strategic innovation.

[0063] In the present invention, all the retrieval involved is implemented by a search engine (such as the Elastic Search engine); the same applies below.

[0064] The embodiment of the present invention provides a method for assisting data analysis and private domain knowledge entry based on LLM, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of the method for assisting data analysis and private domain knowledge entry based on LLM is shown. The processing flow of the method may include the following steps:

[0065] S1. Preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking several knowledge elements;

[0066] S2, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements;

[0067] S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression;

[0068] S4. The LLM Agent performs knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result.

[0069] The present invention aims at the application of LLM (Large Language Model) in the field of data analysis, and proposes a set of general data analysis private domain knowledge entry auxiliary methods. Based on the general prompt word engineering skills (knowledge standard element framework) and the specific expression paradigm of domain knowledge, for each domain knowledge that is effective in the data analysis generation task, it can directly infer the specific logic to be used when querying according to user questions or needs. It can help humans to complete the pre-entry of knowledge efficiently and with high quality. This knowledge will eventually be given to LLM as part of the prompt through the use of RAG, so the relevant knowledge content with clearer expression and more self-consistent logic can ensure that LLM can more accurately complete data analysis tasks such as SQL generation.

[0070] The following will be combined Figure 2 The timing diagram shown is used to describe in detail the implementation principle and application of each step of the present invention.

[0071] The present invention mainly involves: the collection of original knowledge texts (and also involves the subsequent provision of data table information for rewriting and optimizing knowledge data), element completion and clarification, expression logic optimization and knowledge integration (storage).

[0072] In order to improve the efficiency of prompt word entry, each piece of domain knowledge that is effective for data analysis generation tasks can directly infer the specific logic to be used when querying based on user questions or needs. The present invention proposes a set of general data analysis private domain knowledge entry auxiliary methods, based on general prompt word engineering techniques (knowledge standard element framework) and specific expression paradigms of domain knowledge, for each piece of domain knowledge that is effective for data analysis generation tasks, can directly infer the specific logic to be used when querying based on user questions or needs. It can help humans to complete the pre-entry of knowledge efficiently and with high quality.

[0073] Furthermore, the method for constructing the knowledge standard element framework includes:

[0074] Collect some data to analyze private domain knowledge;

[0075] Parsing the data to analyze private domain knowledge and obtain knowledge elements in different types of knowledge content;

[0076] Define the trigger words, query logic and applicable scope of each type of knowledge element, and bind them with the type of corresponding knowledge element to generate a data analysis private domain knowledge system;

[0077] The template design component is used to design and generate the knowledge standard element framework based on the data analysis private domain knowledge system.

[0078] like Figure 3 The knowledge standard element framework shown. The knowledge standard element framework set for the knowledge text in the present invention includes trigger words (optional, such as "growth rate" in "when mentioning growth rate"), the query logic to be used (including screening conditions, grouping methods, sorting principles, numerical formats, etc., such as "using percentages to express (this is a count-looking logic, and also a type of query logic)", or a complete indicator definition, indicator caliber, data table definition, data table application method), and scope of application (optional, that is, applicable topics or specific data tables). When no trigger condition is specified, the query logic must be followed by default. Similarly, when the scope of application is not clear, it is applied to all topics / tables by default.

[0079] The steps for constructing a knowledge check template are as follows:

[0080] 1. Clarify the knowledge system:

[0081] Determine the categorization system of knowledge, including the concepts and entities under each category.

[0082] Define the properties of concepts and entities, as well as the semantic relationships between them.

[0083] 2. Design template structure:

[0084] Design the template structure according to the knowledge system to ensure that the template covers all key knowledge points.

[0085] The template should be extensible to facilitate the subsequent addition of new knowledge points.

[0086] 3. Build a question template:

[0087] A large number of question templates with variables (the trigger words of different knowledge elements, the query logic and the applicable scope) are constructed for querying the structured database.

[0088] The template should form a query expression for specific knowledge points in order to generate accurate answers.

[0089] 4. Optimization and maintenance:

[0090] Proofread and maintain the template regularly to ensure its accuracy and effectiveness.

[0091] Research methods for automatically generating templates to overcome the time-consuming and labor-intensive problem.

[0092] Through the above methods, an efficient and accurate knowledge checking template can be constructed to help users quickly locate and check key points in the knowledge system.

[0093] Based on the above method, the present invention constructs a template, and can construct a knowledge standard element framework (or a knowledge standard element template) based on a zero-code or low-code construction method, such as: selecting a suitable zero-code or low-code platform; determining the template type and functional requirements; searching and selecting pre-made templates on the platform; adjusting the template layout and style according to requirements; adding or modifying data sources and logic; testing to ensure that the template functions correctly; exporting or publishing the template for use. The template design component (or template design platform) is also used to build the framework, and the template can also be built through Python. By defining the elements of the corresponding knowledge elements, the knowledge standard element framework is designed and generated according to the data analysis private domain knowledge system, and the corresponding knowledge standard element template is automatically generated.

[0094] After construction, the framework is configured on the LLM Agent to perform the following optimization checks according to various pre-defined logics (see the following description for details).

[0095] Combination Figure 2 As shown, the input detection is mainly carried out on the following three aspects of knowledge elements.

[0096] 1. Completion and clarification of knowledge elements

[0097] Furthermore, S2, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements, including:

[0098] Obtain original knowledge text and pre-prepared data table information;

[0099] The LLM Agent identifies several knowledge elements in the original knowledge text based on the knowledge standard element framework, further extracts several keywords from the knowledge elements, and identifies keywords with missing fields;

[0100] Use a search engine to retrieve data knowledge related to the keyword with missing fields: data table name, field name and enumeration value;

[0101] The LLM Agent combines the data table information and the retrieved data knowledge to rewrite and complete the knowledge elements of the keywords with missing fields;

[0102] The rewritten and completed knowledge statement is output and sent to the user end, and the administrator confirms / corrects it, and obtains the knowledge statement with complete elements.

[0103] Furthermore, when rewriting and completing the knowledge elements of the keywords with missing fields, it also includes:

[0104] If there are knowledge elements that cannot be rewritten or completed, a marking symbol is called to mark the place where the knowledge elements that cannot be rewritten or completed are placed.

[0105] Here, LLM is mainly used to check whether the knowledge elements in the original knowledge sentence are complete and unambiguous, and automatically complete and clarify them for the user to confirm whether the rewrite is correct.

[0106] The main steps of identifying knowledge elements in knowledge text by LLM Agent are as follows:

[0107] 1. Perception and data collection

[0108] ‌Perception System‌: LLM Agent collects text information through the perception system, which is the first step in its interaction with the environment.

[0109] 2. Data preprocessing and storage

[0110] ‌External storage‌: In a multi-round conversation, LLM Agent stores the conversation of each interaction in external storage for subsequent use.

[0111] 3. Context retrieval and merging

[0112] ‌Retrieve relevant information‌: In the next round of dialogue, the LLM Agent retrieves information related to the target question from external storage as context.

[0113] ‌Merge context‌: Merge the retrieved context information into the prompt word of the question and send it to the big model for processing.

[0114] 4. Knowledge element identification and reasoning

[0115] ‌Utilizing LLM‌: LLM Agent uses its core large language model to identify and reason about knowledge elements, thereby understanding text data and extracting key information.

[0116] In this embodiment, LLM Agent screens the identified knowledge elements and their keywords based on the definitions such as trigger words, query logic and applicable scope in the knowledge standard element framework. This process ensures that LLM Agent can effectively identify and extract key knowledge elements from the knowledge text.

[0117] The completion and clarification of knowledge elements (or rewriting and completion) mainly involves the following four aspects:

[0118] 1. Keyword extraction: Based on the preset knowledge standard element framework, LLM Agent is used to identify and extract relevant elements in the original knowledge sentence. For example, for the sentence "When online is mentioned, filter e-commerce channels", the trigger condition is "When online is mentioned", the trigger word is "online", and the query logic corresponding to this condition is "filter e-commerce channels", which is a simple filtering condition. The filtering enumeration value is "e-commerce channels", but the filtering field is missing. The scope of application is not explicitly mentioned here and is also marked as missing.

[0119] 2. Data knowledge retrieval: For some keywords in the above elements, use the search engine to retrieve the relevant data table names, field names and enumeration values. This is collectively called data knowledge, and its text and semantic vectors (generated based on the LLM Embedding model) have been pre-stored. The retrieval results are used for the next step of RAG (retrieval enhancement generation).

[0120] 3. Element completion and rewriting: When "e-commerce channels" or similar words are found in multiple different fields in step 2, the LLM Agent is required to combine the search results and the basic information of the data table (such as field annotations, etc.) to select one of the fields to rewrite and complete the knowledge sentence, and the same applies to other elements. For missing elements that cannot be completed, use specific symbols to mark the placeholders. The application will present the sentences before and after the rewrite to the expert user for quick confirmation, supplementation and correction. If the user completes the confirmation directly, it will proceed to the next step, otherwise it is necessary to re-execute steps 1 to 3 for the edited sentence.

[0121] 4. Unified symbol rewriting: Using LLM Agent, combined with the keyword extraction results, data table information, and related data knowledge in the previous steps, use unified punctuation marks to mark the entity words (including fields, enumeration values, business concept words, etc.) mentioned in the statement. This is mainly based on mainstream SQL syntax such as MySQL, using backticks to mark table names and field names, single quotes to mark enumeration values, and double quotes to mark other key concept words that need to be emphasized. By simply adding these punctuation marks, the necessary information dimensions can be introduced without adding additional understanding burden, which helps to improve the performance of SQL generation tasks.

[0122] 2. Knowledge Representation Logic Optimization

[0123] Further, S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression, including:

[0124] The LLM Agent identifies and extracts the knowledge elements and query logic in the original knowledge text based on the knowledge standard element framework;

[0125] Determine whether the original knowledge text contains the knowledge elements that are not core elements: if so, determine that the expressions are redundant and delete the knowledge elements that are not core elements; otherwise, abandon them;

[0126] Determine whether the query logic in the original knowledge text is the default logic under the common data analysis requirements in the subject; if so, optimize the expression of the query logic and put the default logic in front; otherwise, abandon it;

[0127] Output the clearly expressed knowledge text obtained after the above operation.

[0128] Further, S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression, and also includes:

[0129] The LLM Agent identifies and extracts the knowledge expression logic and various relational association words in the original knowledge text;

[0130] Based on the query logic in the code instance defined in the knowledge standard element framework, the knowledge expression logic in the original knowledge text is logically checked: if the logic is inconsistent, the knowledge expression logic in the original knowledge text is logically rewritten using the query logic in the code instance; otherwise, it is abandoned;

[0131] Based on the query logic of the association relationship defined in the knowledge standard element framework, rewriting the association words of various relationships in the original knowledge text;

[0132] Output the clearly expressed knowledge text obtained after the above operation.

[0133] Here we mainly use LLM to check whether the logical expression of knowledge statements with complete elements is clear enough, unambiguous, and conforms to the most common expression habits to ensure strong readability and easy understanding. The main logic optimization includes the following optimization solutions:

[0134] 1. Redundancy elimination: LLM Agent determines whether there are redundant expressions in the knowledge statement except for the core elements (non-core elements and core elements. In the query logic of the definition framework, the indicator definition or indicator caliber or data table definition can be used to define which knowledge elements are non-core elements and which are core elements. Therefore, LLMAgent can identify the core and non-core based on the framework definition) according to the knowledge standard element framework, and eliminates and rewrites them (such as deleting the sub-statement "no need to use XXX logic") to avoid unnecessary interference.

[0135] In the present invention, according to the predefined logical structure, the LLM Agent can automatically complete the corresponding logic or elements rewriting according to the definition in the framework for the logic or elements detected by the LLM Agent that need to be rewritten. Therefore, there is no need for manual participation by the user, and there is no need to manually modify the original logic or elements in the knowledge sentence according to the defined logic, which greatly improves the efficiency of inputting knowledge prompt words. The same below.

[0136] Simplified rewriting: Use LLM Agent to identify whether the query logic in the statement is the default logic, such as "When it is explicitly required to include invalid orders, keep the invalid orders, otherwise remove them." Use more general and concise sentences to explain the query logic that should be used by default (LLM Agent can rewrite the logic in the original statement according to the default query logic in the framework), and under what special trigger conditions (if any) it will not be used, such as "invalid orders are removed by default unless the user asks to explicitly include them."

[0137] 3. Coding rewriting: Use LLM Agent to adjust the word order, give common code style knowledge expression examples (query logic in code examples) for reference, and rewrite using expression structures closer to the code style to ensure that the logic is clearer and more concise, and more naturally adaptable to LLM capabilities. The code example contains the code style of the corresponding task, that is, the word order of code execution, and the LLM Agent can read the word order logic of the code word order in the code example provided. Therefore, the word order logic of the code word order can be used to logically rewrite the knowledge expression logic in the original knowledge text to adapt to the reading and writing capabilities of LLM. 4. Optimization of associated words: LLM Agent is used to optimize the rewriting of various conditional relationships, parallel relationships, transitional relationships, etc. involved in the sentence, giving priority to the use of several specified associated words (the query logic of different associated relationships is defined in the framework, and several associated words are defined and configured for different associated relationships, so that the LLM Agent can identify the associated relationships of the sentence and call the corresponding associated words in the framework for rewriting). This ensures that the rewritten sentence has a more unified style and a more stable rewriting effect while ensuring that the logic remains unchanged, and can also reduce the overall difficulty of understanding multiple knowledge sentences in the end.

[0138] 3. Integration of related knowledge and explanation of conflicts

[0139] Further, S4, the LLM Agent performs knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result, including:

[0140] Enter the clearly expressed knowledge text;

[0141] The LLM Agent searches the original knowledge base to see whether there are knowledge statements with the same or similar logic as the clearly expressed knowledge text:

[0142] If so, integrate the clearly expressed knowledge text into knowledge sentences with the same or similar logic;

[0143] Otherwise, it is marked as a knowledge statement to be optimized, and the LLM Agent is used to retrieve and mark the association relationship in the knowledge statement to be optimized. Based on the marked association relationship, the LLM Agent rewrites and integrates the knowledge statement to be optimized based on the retrieved knowledge statements with the same or similar logic; if there is a knowledge statement to be optimized whose association relationship is a logical conflict, the corresponding logical conflict marking result is output;

[0144] The knowledge integration result is output to the user end, and is confirmed / corrected by the administrator, and the original knowledge base is updated based on the feedback confirmation / correction result.

[0145] This section is mainly for newly entered knowledge. If there is content in the existing knowledge that is closely related to its logic, completing the knowledge integration in time during the entry stage will significantly improve the application of knowledge in the final LLM and reduce the difficulty of knowledge maintenance. However, there will also be logical conflicts. Conflicting knowledge will seriously interfere with the generation of LLM answers, resulting in a high error rate and instability, which also needs to be detected and prompted to the user. It mainly includes the following integration and conflict detection instructions:

[0146] 1. Knowledge sentence retrieval: For the knowledge sentences to be optimized, first search and match other knowledge with similar semantics. A dual-engine method combining vector retrieval and keyword retrieval is used here to maximize the recall rate. Because although the pure vector retrieval based on the LLM Embedding Model has high matching accuracy, it cannot recall related but logically conflicting (that is, some semantically opposite) knowledge in this scenario. Keyword retrieval is not a traditional method based on word segmentation and word frequency statistics, but is based on the "trigger word", "field", "enumeration value" and other types of keywords extracted in the previous element completion link for retrieval and matching, which effectively improves the recall rate and accuracy of the final retrieval results. (The knowledge base stores the text, keywords and semantic vectors of all entered knowledge.)

[0147] 2. Relationship labeling: Use LLM Agent to determine the relationship between each piece of knowledge content in the search results and the statement K to be optimized, including three situations: "logically closely related", "logically irrelevant", and "logically conflicting". If it is a "logical conflict", it is necessary to explain what the conflict point is.

[0148] 3. Integration and rewriting: For all knowledge items marked as "logically closely related" (including statement K), first use LLM Agent to rewrite and integrate. For example, statement 1 is "When mentioning "brand", it is necessary to filter "brand" is not null", and statement 2 is "When the query does not involve "brand", it is necessary to restrict "brand" is null", rewrite it to "default use condition ("brand" isnull), but when "brand" is explicitly mentioned, use condition ("brand" is not null)".

[0149] 4. Manual confirmation: Several knowledge statements before and after integration are presented to expert users through an interactive interface, and the experts confirm whether the rewriting plan is approved. Among them, if there is existing knowledge with "logical conflict" in step 2, the knowledge items and the reasons for the conflict must also be presented together, and the experts will complete the final revision.

[0150] Therefore, based on the implementation of the present invention, the general data analysis private domain knowledge entry auxiliary method of the present invention can be used, based on the general prompt word engineering skills (knowledge standard element framework) and the specific expression paradigm of domain knowledge, for each domain knowledge that is effective for the data analysis generation task, it can directly infer the specific logic to be used when querying according to user questions or needs. It can help humans to complete the pre-entry of knowledge efficiently and with high quality. This knowledge will eventually be given to LLM as part of the Prompt through the use of RAG, so the relevant knowledge content with clearer expression and more self-consistent logic can ensure that LLM can more accurately complete data analysis tasks such as SQL generation.

[0151] Figure 4 1 is a block diagram of a data analysis private domain knowledge entry assisting device based on LLM according to an exemplary embodiment, and the device is used for a data analysis private domain knowledge entry assisting method based on LLM. Figure 4 The device includes an element template construction module 310, an element completion and clarification module 320, an expression logic optimization module 330 and a knowledge integration and update module 340. Among them:

[0152] An element template construction module is used to preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking a number of knowledge elements;

[0153] An element completion and clarification module is used for the LLM Agent to complete and clarify the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements;

[0154] The expression logic optimization module is used for the LLM Agent to rewrite and optimize the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression;

[0155] The knowledge integration and updating module is used for the LLM Agent to perform knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and update the original knowledge base based on the integration result.

[0156] Please understand and implement the functions and interactive relationships of the element template construction module 310, the element completion and clarification module 320, the expression logic optimization module 330 and the knowledge integration and update module 340 according to the above method steps, which will not be repeated here.

[0157] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include the above Figure 4 The LLM-based data analysis private domain knowledge entry auxiliary device is shown. Optionally, the electronic device 410 may include a first processor 2001.

[0158] Optionally, the electronic device 410 may further include a memory 2002 and a transceiver 2003 .

[0159] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0160] Combine the following Figure 5 The components of the electronic device 410 are described in detail:

[0161] The first processor 2001 is the control center of the electronic device 410, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0162] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0163] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0164] In a specific implementation, as an embodiment, the electronic device 410 may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0165] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0166] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently and access the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0167] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0168] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0169] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0170] It should be noted that Figure 5 The structure of the electronic device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0171] In addition, the technical effects of the electronic device 410 can refer to the technical effects of the LLM-based data analysis private domain knowledge entry auxiliary method described in the above method embodiment, and will not be repeated here.

[0172] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0173] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0174] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0175] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0176] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0177] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0178] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0184] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A data analysis private domain knowledge entry auxiliary method based on LLM, characterized in that: The method comprises: S1. Preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking a number of knowledge elements; the method for constructing the knowledge standard element framework includes: Collect some data to analyze private domain knowledge; Parsing the data to analyze private domain knowledge and obtain knowledge elements in different types of knowledge content; Define the trigger words, query logic and applicable scope of each type of knowledge element, and bind them with the type of corresponding knowledge element to generate a data analysis private domain knowledge system; Using a template design component, designing and generating the knowledge standard element framework according to the data analysis private domain knowledge system; S2, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements; S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression; S4. The LLM Agent performs knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result.

2. The LLM-based data analysis private domain knowledge entry auxiliary method according to claim 1 is characterized in that: S2. Based on the knowledge standard element framework, the LLM Agent completes and clarifies the knowledge elements in the original knowledge text to obtain knowledge sentences with complete elements, including: Obtain original knowledge text and pre-prepared data table information; The LLM Agent identifies several knowledge elements in the original knowledge text based on the knowledge standard element framework, further extracts several keywords from the knowledge elements, and identifies keywords with missing fields; Use a search engine to retrieve data knowledge related to the keyword with missing fields: data table name, field name and enumeration value; The LLM Agent combines the data table information and the retrieved data knowledge to rewrite and complete the knowledge elements of the keywords with missing fields; The rewritten and completed knowledge statement is output and sent to the user end, and the administrator confirms / corrects it, and obtains the knowledge statement with complete elements.

3. The LLM-based data analysis private domain knowledge entry auxiliary method according to claim 2 is characterized in that: When rewriting and completing the knowledge elements of the keywords with missing fields, it also includes: If there are knowledge elements that cannot be rewritten or completed, a marking symbol is called to mark the place where the knowledge elements that cannot be rewritten or completed are placed.

4. The LLM-based data analysis private domain knowledge entry auxiliary method according to claim 1 is characterized in that: S3. Based on the knowledge standard element framework, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements to obtain a knowledge text with clear expression, including: The LLM Agent identifies and extracts the knowledge elements and query logic in the original knowledge text based on the knowledge standard element framework; Determine whether the original knowledge text contains the knowledge elements that are not core elements: if so, determine that the expressions are redundant and delete the knowledge elements that are not core elements; otherwise, abandon them; Determine whether the query logic in the original knowledge text is the default logic under the common data analysis requirements in the subject; if so, optimize the expression of the query logic and put the default logic in front; otherwise, abandon it; Output the clearly expressed knowledge text obtained after the above operation.

5. The LLM-based data analysis private domain knowledge entry auxiliary method according to claim 4 is characterized in that: S3, the LLM Agent rewrites and optimizes the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression, and also includes: The LLM Agent identifies and extracts the knowledge expression logic and various relational association words in the original knowledge text; Based on the query logic in the code instance defined in the knowledge standard element framework, the knowledge expression logic in the original knowledge text is logically checked: if the logic is inconsistent, the knowledge expression logic in the original knowledge text is logically rewritten using the query logic in the code instance; otherwise, it is abandoned; Based on the query logic of the association relationship defined in the knowledge standard element framework, rewriting the association words of various relationships in the original knowledge text; Output the clearly expressed knowledge text obtained after the above operation.

6. The LLM-based data analysis private domain knowledge entry auxiliary method according to claim 4 is characterized in that: S4. The LLM Agent integrates the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and updates the original knowledge base based on the integration result, including: Enter the clearly expressed knowledge text; The LLM Agent searches the original knowledge base to see whether there are knowledge statements with the same or similar logic as the clearly expressed knowledge text: If so, integrate the clearly expressed knowledge text into knowledge sentences with the same or similar logic; Otherwise, it is marked as a knowledge statement to be optimized, and the LLM Agent is used to retrieve and mark the association relationship in the knowledge statement to be optimized. Based on the marked association relationship, the LLM Agent rewrites and integrates the knowledge statement to be optimized based on the retrieved knowledge statements with the same or similar logic; if there is a knowledge statement to be optimized whose association relationship is a logical conflict, the corresponding logical conflict marking result is output; The knowledge integration result is output to the user end, and is confirmed / corrected by the administrator, and the original knowledge base is updated based on the feedback confirmation / correction result.

7. An LLM-based data analysis private domain knowledge entry auxiliary device, the LLM-based data analysis private domain knowledge entry auxiliary device is used to implement the LLM-based data analysis private domain knowledge entry auxiliary method as claimed in any one of claims 1 to 6, characterized in that: The device comprises: An element template construction module is used to preset and construct a knowledge standard element framework and configure it in the LLM Agent, wherein the knowledge standard element framework includes trigger words, query logic and applicable scope for checking a number of knowledge elements; An element completion and clarification module is used for the LLM Agent to complete and clarify the knowledge elements in the original knowledge text based on the knowledge standard element framework to obtain a knowledge sentence with complete elements; The expression logic optimization module is used for the LLM Agent to rewrite and optimize the expression logic in the knowledge sentence with complete elements based on the knowledge standard element framework to obtain a knowledge text with clear expression; The knowledge integration and updating module is used for the LLM Agent to perform knowledge integration on the clearly expressed knowledge text based on the knowledge sentences in the original knowledge base, and update the original knowledge base based on the integration result.

8. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images