Method for automatically extracting question number data from heat supply ERP system by using large model

Abstract

The invention provides a method for automatically extracting question data from a heat supply ERP system by using a large model, and belongs to the technical field of heat supply management. The method comprises the following steps: automatically extracting index-SQL template mapping data from a system back-end source code based on LLM and RAG technologies; inputting the system screenshot, the index list and the professional knowledge base into a multi-modal large model, and automatically generating a question and answer pair set containing indexes, numerical values, time and questions and answers; for each question and answer pair, calling LLM to instantiate a corresponding SQL template into an executable statement, checking the consistency of a result and an answer through the LLM after execution, and automatically adjusting a wrong SQL to generate an accurate question-SQL-answer triple; and further in combination with scores of service experts on candidate SQL, differences are analyzed by using LLM, experience is summarized, an expert knowledge base is dynamically updated, and SQL generation quality is continuously optimized. According to the method, the manual marking cost can be greatly reduced, and high-quality training data is provided for the Text-to-SQL model in the heat supply field.

Description

Technical Field

[0001] This invention relates to a method for automatically extracting query data from a heating ERP system using a large model, belonging to the field of heating management technology. Background Technology

[0002] Text-to-SQL technology refers to the technique of converting users' natural language questions into SQL statements for querying a database. A question-and-answer system is a software system that includes a front-end user interface, back-end services, and a database system. Users ask questions in natural language on the interface, the system uses text-to-SQL technology to generate SQL statements, queries the database, and then organizes the database results into a natural language answer, which is displayed to the user on the front end. The core capability of a question-and-answer system is text-to-SQL, and the accuracy of text-to-SQL directly determines the accuracy of the system's responses.

[0003] In the heating sector, user inquiries in query systems are concentrated in the heating production field. These inquiries typically involve various operational and statistical data related to facilities and equipment such as heat source plants, heat exchange stations, heating networks, boilers, and generating units. This involves numerous database tables and complex SQL queries, making the implementation of text-to-sql technology in the heating sector quite challenging. One technical approach is to use a large language model combined with industry knowledge and data table knowledge in the heating sector to develop prompt words and then call the large language model to implement text-to-sql. Considering the slow inference speed of large-parameter large language models and the extremely high computing resources required for deploying them, the company prefers to use a small-parameter large language model (MOE architecture) for fine-tuning the text-to-sql capabilities in the heating sector. This improves text-to-sql capabilities while ensuring fast inference speed and low computing resource requirements, making it a more ideal solution.

[0004] However, this approach requires a large amount of high-quality data on heating issues as training data for model fine-tuning. Since manual annotation is too costly, the core challenge of this approach is to generate high-quality data as automatically as possible with minimal human intervention. Summary of the Invention

[0005] The purpose of this invention is to provide a method for automatically extracting query data from a heating ERP system using a large model, which can greatly reduce the cost of manual annotation and provide sufficient and reliable training, testing and validation data for fine-tuning Text-to-SQL models in the heating field.

[0006] To achieve the above objectives, the present invention employs the following technical solution: A method for automatically extracting query data from a heating ERP system using a large model includes the following steps: Based on the Large Language Model (LLM) and Retrieval Enhancement Generation (RAG) technology, SQL templates corresponding to the indicators displayed on the front-end page are automatically extracted from the back-end service source code of the heating ERP system to form the indicator-SQL template mapping data. Read the indicator list and heating professional knowledge base from the mapping data of the indicator-SQL template; input the screenshot of each heating ERP system front-end page, indicator list and professional knowledge base into the multimodal large model to generate a question-and-answer pair data set containing indicators, indicator values, time information, questions and answers; Read the question-answer pair data set and the mapping data of the indicator-SQL template; for each question-answer pair, locate the corresponding indicator and SQL template according to its question, and call the large language model to instantiate the SQL template into an executable SQL statement; The SQL statement is executed to obtain the query results, and the large language model is called to verify the consistency between the query results and the answers in the question-answer pair; the SQL statements that fail the verification are adjusted, and finally an accurate question-SQL-answer triplet dataset is generated. Based on the feedback scores of business experts on candidate SQL statements, the differences between SQL statements with different scores are analyzed through a large language model. The generation experience is summarized and the expert experience base is dynamically updated. The updated expert experience base is then used to optimize the generation quality of subsequent SQL statements.

[0007] Preferably, the specific methods for forming the indicator-SQL template dictionary include: Obtain the backend service source code of the heating ERP system, and establish indicator-backend interface description data for the system frontend business indicators and backend service interface descriptions. Construct a retrieval enhancement generation system based on the source code of the backend service, and slice, vectorize, and store the source code in a vector database; The system iterates through the backend interface description data of the metrics, and for each business metric and its corresponding interface description, retrieves the relevant code snippets through the retrieval enhancement generation system. The business metrics, interface descriptions, and related code snippets are input into the large language model, and the designed prompt words guide the large language model to output the SQL query template corresponding to the business metrics. The output SQL query templates are reviewed and corrected to form the mapping data between indicators and SQL templates.

[0008] Preferably, the specific methods for generating the question-and-answer dataset include: Collect screenshots of data from different business pages of the heating ERP system at different time ranges; Input the data screenshot, the list of business indicators in the mapping data of the indicator-SQL template, and the professional knowledge base of the heating industry into the multimodal large language model; The multimodal large language model is guided by designed prompts to identify and extract indicators, indicator values, time information, natural language questions and natural language answers that are aligned with the business indicator list from the screenshot, and outputs a set of question-answer pairs generated based on the information in a structured format.

[0009] Preferably, adjusting the SQL statement that failed the verification includes data repair and data error correction; If the SQL statement fails to execute or the result is empty, perform data repair. If the SQL statement executes successfully but the result is inconsistent with the answer, then data correction is performed.

[0010] Preferably, the data error correction process specifically includes: Obtain the execution result of the SQL statement and perform validation. Apply validation prompts, including the question, answer, execution result, and SQL statement format. Call the large language model to generate the result. The prompts require the large language model to determine whether the core data of the answer and the generated result are consistent. Analyze the logic of the generated SQL statement and generate the judgment result and judgment basis. If the result is consistent, the triple consisting of the question, SQL statement, and answer is stored in the success list; if the result is inconsistent, data correction is performed. Apply data correction prompts, including the question, metric, SQL template, SQL generation specification, SQL statement format, answer, and error result; use a large language model to analyze the differences between the answer and the error result, and generate a corrected SQL statement; call the large language model to regenerate the SQL statement format, and execute it to get a new result; if the new result is still empty or the execution fails, repeat this step until the set number of times is reached and the result remains unchanged, then store the question, metric, SQL template, SQL generation specification, and SQL statement format in the failure list.

[0011] Preferably, the data repair process specifically includes: Apply the prompts for result verification, call the large language model to generate results, use the large language model to determine whether the core data of the file and the result are consistent, and analyze the logic of the generated SQL statement to generate judgment results and judgment basis; If the result is consistent, the triple consisting of the question, SQL statement, and answer is stored in the success list; if the result is inconsistent, data correction is performed.

[0012] Preferably, the step of analyzing the differences between SQL statements with different scores based on feedback scores from business experts, summarizing generation experience and dynamically updating the expert experience base, and using the updated expert experience base to optimize the generation quality of subsequent SQL statements specifically includes: Collect issues from the heating ERP system, establish an initial issue list, and for each issue in the initial issue list, generate two or more candidate SQL statements as candidate groups based on the heating professional knowledge base and database description knowledge base through a large language model. Obtain the scores or directly labeled correct SQL statements from business experts regarding the candidate SQL statements; Based on the preset group retention strategy, candidate groups that contain both high-scoring and low-scoring SQL statements are selected. For the candidate group, the differences between high- and low-scoring SQL statements are analyzed using a large language model, and the experience in generating SQL for this problem is summarized. By using a large language model to determine the logical relationship between the summarized experience and the existing experience in the expert experience base, the expert experience base can be updated, deleted, or modified online to achieve online updates and enhancements.

[0013] Preferably, the group retention strategy is as follows: a group is retained if both of the following conditions are met: at least one SQL statement in the candidate group has a score of 1 point and at least one SQL statement has a score of 0 points; and not all SQL statements in the candidate group have a score of 1 point and not all have a score of 0 points.

[0014] The advantages of this invention are as follows: By employing automated methods and utilizing large language models and multimodal large models, this invention directly extracts and generates high-quality query data from the heating ERP system, significantly reducing the workload and cost of manual annotation. The data covers all aspects of the heating industry, meeting the need for high-quality training data for fine-tuning the text2sql model.

[0015] By fine-tuning the text-to-sql model using the high-quality query data generated by this invention, the model's text-to-sql conversion capability is significantly improved. This enhances the accuracy of the query system's responses, enabling users to obtain the necessary heating production data more quickly and accurately, thereby improving the system's usability and user experience.

[0016] This invention designs an online reinforcement learning module that utilizes feedback from business experts. By receiving review comments and annotated SQL from business experts, it generates expert experience and knowledge, which is then automatically incorporated into the data generation process. This not only improves the quality of data generation but also enables the automated accumulation and application of expert experience and knowledge, providing strong support for continuous system optimization. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1 like Figure 1 As shown, a method for automatically extracting query data from a heating ERP system using a large model includes the following steps: S1: Based on the Large Language Model (LLM) and Retrieval Enhancement Generation (RAG) technology, the SQL templates corresponding to the indicators displayed on the front-end page are automatically extracted from the back-end service source code of the heating ERP system to form the indicator-SQL template mapping data. S2: Read the indicator list and heating professional knowledge base from the mapping data of the indicator-SQL template; input the screenshot of each heating ERP system front-end page, indicator list and professional knowledge base into the multimodal large model to generate a question-and-answer pair data set containing indicators, indicator values, time information, questions and answers; S3: Read the question-answer pair data set and the mapping data of the indicator-SQL template; for each question-answer pair, locate the corresponding indicator and SQL template according to its question, and call the large language model to instantiate the SQL template into an executable SQL statement; S4: Execute the SQL statement to obtain the query results, and call the large language model to perform consistency verification between the query results and the answers in the question-answer pair; adjust the SQL statements that fail the verification, and finally generate an accurate question-SQL-answer triplet dataset; S5: Based on the feedback scores of business experts on candidate SQL statements, the differences between SQL statements with different scores are analyzed through a large language model, the generation experience is summarized and the expert experience base is dynamically updated, and the quality of subsequent SQL statement generation is optimized by using the updated expert experience base.

[0021] As a refinement of the above embodiment, step S1 first requires the company's internal developers to provide the backend service source code of the ERP system. The developers also need to explain how the various metrics displayed on the front-end page are calculated in the backend service code, where the SQL templates are stored, and how the code instantiates SQL query statements from the SQL templates based on the interface parameters, etc. With this knowledge, targeted prompts and the RAG system can be designed, and an automatic generation program for metric-SQL template data can be developed.

[0022] Specific methods include: S101: Obtain the backend service source code of the heating ERP system and establish indicator-backend interface description data between the system frontend business indicators and the backend service interface descriptions; taking the "Heat Exchange Station Energy Consumption Report" as an example, the mapping is as follows: {“Heat Station Energy Consumption Report”: “The interface address is / heat / station / consumption / getReport, and the function signature is public HeatStationConsumeReportRespVo getReport(@ValidHeatStationConsumeReportReq req)”} S102: Construct a retrieval enhancement generation system based on the source code of the backend service, slice the source code, vectorize it, and store it in a vector database; vector_db = VectorDatabase() code_chunks = chunk_code_files(erp_backend_codebase) vector_db.ingest(code_chunks) S103: Traverse the metric-backend interface (item-interface-dict) description data, and for each business metric and its corresponding interface description, retrieve the relevant code snippets through the retrieval enhancement generation system; Iterate through each metric in the `item-interface-dict`, including the metric item and the backend interface description (`interface-description`), and retrieve the corresponding code snippets (`relevant_code_snippets`) using RAG. S104: Input the business metrics, interface descriptions, and related code snippets into the large language model, and guide the large language model to output the SQL query template corresponding to the business metrics through the designed prompt words; LLM uses DeepSeek V3.1 deployed on the company's intranet; S105: Review and correct the output SQL query template to form the mapping data between indicators and SQL templates.

[0023] As a refinement of the above embodiments, the specific method for generating the question-and-answer pair data set in step S2 includes: S201: Collect screenshots of data from different business pages of the heating ERP system at different time ranges; For example, open a page in the heating ERP system, select a time range, and click the search button to display energy consumption reports for heat exchange stations within that time range, including indicators such as heat consumption, electricity consumption, and unit heat consumption. Take a screenshot of this page and save it as an image file. You can drag the page up and down to display more heat exchange stations, or drag it left and right to display more indicators, thus capturing more images. You can also change the time range, click "search" to display data for other time ranges, and then take screenshots. You can also switch to different pages to obtain screenshots of other indicators, as shown in the image below.

[0024] S202: Input the data screenshot, the list of business indicators in the mapping data of the indicator-SQL template, and the professional knowledge base of the heating industry into the multimodal large language model; S203: Guide the multimodal large language model to identify and extract indicators, indicator values, time information, natural language questions and natural language answers that are consistent with the business indicator list from the screenshot through the designed prompt words, and output the question-answer pair data set generated based on the information in a structured format.

[0025] Specifically, the process includes reading the indicator-SQL template dictionary `item-sql-dict`; extracting all indicators from it as `item_list`; reading the heating industry knowledge base `heat_knowledge_base`; reading the screenshot file directory; for each image, applying question-and-answer generation prompts sequentially; calling the multimodal large-scale model to generate results; and parsing the results to obtain several generated question-and-answer pairs. The prompts include the template dictionary `item-sql-dict` and the knowledge base `heat_knowledge_base`. The large-scale model is required to align the indicator names extracted from the images with those in `item_list`, extract the corresponding numerical values ​​for each indicator in the images, and generate natural language questions and answers based on the indicator names and values. The output format is required to be JSON, specifically: json [ {{ "item": "The name of the extracted metric, which must be selected from item_list". "value": "The extracted numerical value corresponding to this indicator must have dimensions and conform to the explanation of this indicator in heat_knowledge_base". "time": "Time information extracted from the image, which may be a time range or a specific point in time." "question": "A natural language question generated based on the name and time information of this indicator". "answer": "Generate a natural language answer based on the corresponding numerical value of the indicator." }}, ] Collect the generation results for each image. Generally, one image will contain multiple results. After deduplication, save them as local files. Each result contains a question, an answer, an indicator, an indicator value, and time information.

[0026] As a refinement of the above embodiments, step S3 specifically includes: S301: Read the question-answer pair data set and extract the question-answer list qa_list from it; at the same time, based on the read item and question information, establish a mapping dictionary query-item-dict from questions to metrics; read the mapping data of metrics-SQL templates (metric-SQL template dictionary item-sql-dict). S302: Iterate through qa_list, retrieve the query and answer, query-item-dict to find the corresponding indicator item, and then query-sql-dict to find the corresponding SQL template sql_template; S303: Use SQL-generated prompts, including the query, the metric item, the SQL template (sql_template), the SQL generation specifications, and call a large language model to generate the results. The prompts require the large model to extract the query elements from the query, such as time and entity name (heat exchange station, heat source, unit, etc.), then instantiate the SQL template, replacing the time and entity information, ensuring the core query metric aligns with the item, and that the generated SQL statement conforms to the specified SQL generation specifications. The output format must be JSON, specifically: json {{ "sql": "The generated SQL statement must conform to MySQL specifications". "describe": "Generates a natural language description of the query logic of the SQL statement, and references the table names and field names that appear in the description." "table_info": Provides the names of all tables involved in the SQL statement, and the names of the fields involved in each table. }} As a refinement of the above embodiments, step S4 specifically includes: S401: Parse the generated results of the large language model, extract the sql, describe, and table_info information; execute the SQL statement and obtain the execution result result. If result is not empty, run the data validation logic and go to S402; if result is empty, or the SQL statement execution fails, run the data repair logic and go to S404.

[0027] S402: If the SQL execution result has a value, validate the result in this step; apply the result validation prompts, including the query, the correct answer, the execution result, and the generated SQL / describe / table_info information; call the large language model to generate the result. The prompts require the large model to determine whether the core data of the answer and result are consistent, ignoring specific format differences, and to analyze the logic of the generated SQL to see if it meets the core requirements of the query. Furthermore, it uses the describe and table_info information to help determine whether the SQL execution process is logically sound and whether the result reflects reasonable output. The output format must be JSON, specifically: json {{ "res": "Judgment result, consistent or inconsistent" "reason": "to provide the basis for the judgment" }}.

[0028] S403: Parse the result generated by S402 and extract the res and reason information. If res is "consistent", the generated SQL is considered to meet the requirements, and a (query, sql, answer) triple is obtained. The triple is collected into the result list success_list, and then the process is transferred to S302 to process the next question and answer; otherwise, the generated SQL is considered to be incorrect, the data correction logic is run, and the process is transferred to S405.

[0029] S404: If the result in S401 is empty, or the SQL statement execution failed, data repair is performed in this step. Apply the data repair prompts, including the problem query, the indicator item, the SQL template sql_template, the SQL generation specifications, the generated sql / describe / table_info information, and the error message (or emptiness) of the execution result. Call the large language model to generate the result. The prompts require the large model to check the SQL statement for problems and errors based on the generated SQL statement, error message, and other relevant information, and to perform targeted repairs. Call the large language model again and parse the result to obtain sql / describe / table_info. Execute the SQL to obtain the result result_new. Process the result_new according to its different cases: If result_new is still empty, or the new SQL statement still fails to execute, update the error message and continue to redirect to S404; if the number of retries exceeds 3, abandon the repair, collect query, item, sql_template, and generated sql / describe / table_info into the failed data list failed_list, and then continue to process the next question and answer; If Result_new is not empty, then proceed to S402 for data validation.

[0030] S405: If data validation fails, correct the generated SQL in this step. The correction logic is similar to S404, except that the prompt words do not contain error information (initially not). A correct answer (`answer`) and an incorrect result (`result`) are added. The large language model is required to analyze the differences between the correct and incorrect results and provide possible corrected SQL statements. The large language model is called again, and the results are parsed to obtain `sql / describe / table_info`. The SQL is executed to obtain the result `result_new`. Different cases of `result_new` are handled accordingly: If result_new is still empty, or the new SQL statement still fails to execute, repeat S405. Note that in this case, to avoid an infinite loop, it will not go to S404. If the number of retries exceeds 3, abandon error correction, collect query, item, sql_template, and generated sql / describe / table_info into the failed data list failed_list, and then go to S302 to process the next question and answer. If result_new is not empty, proceed to S402 for data validation, and record the number of validations already performed. If the number of validations exceeds 3, abandon error correction, collect query, item, sql_template, and generated sql / describe / table_info into the failed data list failed_list, and then proceed to S302 to process the next question and answer.

[0031] S406: After the loop ends, `success_list` contains the successfully generated questions, answers, and SQL triples; `failed_list` contains the questions that failed to generate and related SQL information. The data in `failed_list` can be manually processed to correct the SQL and added back to `success_list`; alternatively, `failed_list` can be discarded without further processing.

[0032] As a refinement of the above embodiments, step S5 specifically includes: S501: Collect issues from the heating ERP system, establish an initial issue list, and for each issue in the initial issue list, generate two or more candidate SQL statements as candidate groups based on the heating professional knowledge base and database description knowledge base through a large language model.

[0033] Specifically, an initial question list (question_list) is created, the heating knowledge base (heat_knowledge_base) and the heating database table description knowledge base (database_knowledge_base) are read, and the expert experience base (expert_experiences) is initialized to empty. The question_list can contain several questions designed by business personnel based on their experience, such as 100 common questions.

[0034] Iterate through each question in the `question_list` and generate five alternative SQL statements (sql1, sql2, sql3, sql4, sql5) based on the current `expert_experiences`, `heat_knowledge_base`, and `database_knowledge_base`. Here, `prompt_gen_sql` is a prompt word for generating SQL candidate statements, requiring the large language model to generate 5 SQL statements that include information from the current `expert_experiences`, `heat_knowledge_base`, and `database_knowledge_base`; `LLM` is the large language model DeepSeek V3.1. The set of currently generated sql1, sql2, sql3, sql4, and sql5 is defined as the current SQL group, denoted as `cur_sql_group`.

[0035] S502: Obtain the scores of the candidate SQL statements or the directly marked correct SQL statements from business experts.

[0036] Specifically, business experts review and annotate SQL statements 1 through 5, primarily in two ways: The scoring mode assigns a score to each SQL statement, divided into three levels: 1 point for completely correct; 0.5 points for partially correct; and 0 points for incorrect. The standard for partially correct is that the core calculation logic of the SQL statement is correct, but some details such as the matching of time conditions and entity names are incorrect.

[0037] In the annotation mode, if all five candidate SQL statements are incorrect, or if a business expert can directly annotate a correct SQL statement based on their own annotation principles, the annotated SQL statement will receive 1 point and be added to the `cur_sql_group`. The remaining candidate SQL statements will then be scored according to the scoring mode. If there are any candidate SQL statements that have not been scored by experts, the large language model will be used to calculate their score. , Among them, prompt_score is the prompt word for calculating the score.

[0038] S503: Based on the preset group retention strategy, filter out candidate groups that contain both high-scoring and low-scoring SQL statements.

[0039] Specifically, the selection of alternative SQL queries depends on the scoring or annotation results of each SQL query in the `cur_sql_group` by business experts. It needs to be determined whether the current `cur_sql_group` should be retained. Different retention strategies can be formulated; for example, one retention strategy could be: In the cur_sql_group, at least one SQL statement has a score of 1 and at least one SQL statement has a score of 0. The SQL scores in cur_sql_group cannot all be 1 points, nor can they all be 0 points; The current group cur_sql_group is retained and the process continues only if both conditions are met; otherwise, the next question is retrieved and the process continues.

[0040] S504: For the candidate group, analyze the differences between high-scoring and low-scoring SQL statements using a large language model, and summarize the experience in generating SQL for this problem.

[0041] Specifically, for each SQL statement in the `cur_sql_group`, the large language model is called to generate a description of the computation process for that SQL statement. , Here, prompt_describe is the prompt word for generating a description of the calculation process, and sql[i] is the i-th sql in cur_sql_group.

[0042] The extraction of expert experience within the group, based on the group retention strategy, involves the presence of both high-scoring and low-scoring SQL statements within the `cur_sql_group`. The large language model then analyzes the reasons for these score differences, ultimately summarizing the group's SQL generation experience. In other words: , Here, `prompt_extract` is the prompt word for experience extraction, `[sql[i]]` refers to the list of SQL statements composed of each SQL statement in `cur_sql_group`, `[describe[i]]` is the description list of the corresponding calculation process, and `[score[i]]` is the score list of the corresponding SQL statement. The prompt word requires the large model to compare the differences between high-scoring and low-scoring SQL statements, and based on the question query and the calculation process of high-scoring and low-scoring SQL statements, analyze which logic in each SQL statement is the scoring point and which logic is the losing point, and summarize the final experience points for generating SQL statements for the question query. These experiences are expressed in natural language and output to `G_experience`.

[0043] S505: Determine the logical relationship between the summarized experience and the existing experience in the expert experience base through a large language model, and add, delete or modify the expert experience base accordingly to realize online updating and enhancement of the experience base.

[0044] Specifically, the expert experience base `expert_experiences` is initially empty, but as the online reinforcement learning algorithm runs, it gradually accumulates a list of experience knowledge in natural language. Based on the newly generated experience `G_experiences` for the current group, the large language model needs to be invoked to analyze how to update `expert_experiences`. Specifically: , Here, `prompt_update` is the prompt word for experience updates. The prompt word requires the large model to use `G_experience` as the standard to determine the logical compatibility between each experience[i] in the `expert_experiences` list and `G_experience`. Based on the compatibility result, it determines whether to delete, modify, or retain `experience[i]`, and further decides on the handling of `G_experience`, such as adding it to `expert_experiences` or not. Finally, based on the analysis results, an update operation list `Update_operations` is output, where each operation contains four types: The ADD type indicates that G_experience can be directly added to expert_experiences. The DELETE type indicates that one experience needs to be output from expert_experiences. In this case, the operation includes the experience to be deleted. The MODIFY type indicates that a single experience in expert_experiences needs to be modified. In this case, the operation includes the src_experience that needs to be modified, as well as the modified mod_experience. The KEEP type indicates that expert_experiences does not need to be updated. Based on the content of each operation in Update_operations, update the expert_experiences list. At this point, the online reinforcement and updating of the expert experience base for the query problem is complete. Proceed to the next problem; note that when generating alternative SQL for the next query, the updated expert experience base is used.

[0045] After processing each question in the `question_list`, the final expert experience base `expert_experiences` is obtained. When generating SQL statements for new questions, this `expert_experiences` is incorporated into the SQL generation prompts to align with expert experience, thus improving the quality of the generated SQL. Repeatedly executing S501-S505 continuously expands the question-SQL dataset, and the `expert_experiences` base also expands, thereby continuously improving the quality of the generated SQL.

[0046] This disclosure also provides an apparatus for automatically extracting query data from a heating ERP system using a large model, including a processor and a memory. Optionally, the apparatus may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can invoke logical instructions in the memory to execute the method described above for automatically extracting query data from a heating ERP system using a large model.

[0047] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0048] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes functional applications and data processing by running the program instructions / modules stored in the memory, thereby realizing the method for automatically extracting data from the heating ERP system using a large model as described in the above embodiments.

[0049] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0050] This disclosure provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described method for automatically extracting query data from a heating ERP system using a large model.

[0051] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0052] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code. It can also be a transient storage medium.

[0053] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically extracting query data from a heating ERP system using a large model, characterized in that, Includes the following steps: Based on the Large Language Model (LLM) and Retrieval Enhancement Generation (RAG) technology, SQL templates corresponding to the indicators displayed on the front-end page are automatically extracted from the back-end service source code of the heating ERP system to form the indicator-SQL template mapping data. Read the indicator list and heating professional knowledge base from the mapping data of the indicator-SQL template; input the screenshot of each front-end page of the heating ERP system, the indicator list and the professional knowledge base into the multimodal large model to generate a question-and-answer pair data set containing indicators, indicator values, time information, questions and answers; Read the question-answer pair data set and the mapping data of the indicator-SQL template; for each question-answer pair, locate the corresponding indicator and SQL template according to its question, and call the large language model to instantiate the SQL template into an executable SQL statement; The SQL statement is executed to obtain the query results, and the large language model is called to perform consistency verification between the query results and the answers in the question-answer pair. Adjust the SQL statements that failed the validation to finally generate an accurate question-SQL-answer triple dataset; Based on the feedback scores of business experts on candidate SQL statements, the differences between SQL statements with different scores are analyzed through a large language model. The generation experience is summarized and the expert experience base is dynamically updated. The updated expert experience base is then used to optimize the generation quality of subsequent SQL statements.

2. The method for automatically extracting query data from a heating ERP system using a large model according to claim 1, characterized in that, The specific methods for forming the indicator-SQL template dictionary include: Obtain the backend service source code of the heating ERP system, and establish indicator-backend interface description data for the system frontend business indicators and backend service interface descriptions. Construct a retrieval enhancement generation system based on the source code of the backend service, and slice, vectorize, and store the source code in a vector database; The system iterates through the backend interface description data of the metrics, and for each business metric and its corresponding interface description, retrieves the relevant code snippets through the retrieval enhancement generation system. The business metrics, interface descriptions, and related code snippets are input into the large language model, and the designed prompt words guide the large language model to output the SQL query template corresponding to the business metrics. The output SQL query templates are reviewed and corrected to form the mapping data between indicators and SQL templates.

3. The method for automatically extracting query data from a heating ERP system using a large model according to claim 1, characterized in that, The specific methods for generating the question-and-answer dataset include: Collect screenshots of data from different business pages of the heating ERP system at different time ranges; Input the data screenshot, the list of business indicators in the mapping data of the indicator-SQL template, and the professional knowledge base of the heating industry into the multimodal large language model; The multimodal large language model is guided by designed prompts to identify and extract indicators, indicator values, time information, natural language questions and natural language answers that are aligned with the business indicator list from the screenshot, and outputs a set of question-answer pairs generated based on the information in a structured format.

4. The method for automatically extracting query data from a heating ERP system using a large model according to claim 1, characterized in that, The adjustment of the SQL statement that failed the verification includes data repair and data error correction; If the SQL statement fails to execute or the result is empty, perform data repair. If the SQL statement executes successfully but the result is inconsistent with the answer, then data correction is performed.

5. The method for automatically extracting query data from a heating ERP system using a large model according to claim 4, characterized in that, The data correction process specifically includes: Obtain the execution result of the SQL statement and perform validation. Apply validation prompts, including the question, answer, execution result, and SQL statement format. Call the large language model to generate the result. The prompts require the large language model to determine whether the core data of the answer and the generated result are consistent. Analyze the logic of the generated SQL statement and generate the judgment result and judgment basis. If the result is consistent, the triple consisting of the question, SQL statement, and answer is stored in the success list; if the result is inconsistent, data correction is performed. Apply data correction prompts, including the question, metric, SQL template, SQL generation specification, SQL statement format, answer, and error result; use a large language model to analyze the differences between the answer and the error result, and generate a corrected SQL statement; call the large language model to regenerate the SQL statement format, and execute it to get a new result; if the new result is still empty or the execution fails, repeat this step until the set number of times is reached and the result remains unchanged, then store the question, metric, SQL template, SQL generation specification, and SQL statement format in the failure list.

6. The method for automatically extracting query data from a heating ERP system using a large model according to claim 5, characterized in that, The data repair process specifically includes: Apply the prompts for result verification, call the large language model to generate results, use the large language model to determine whether the core data of the file and the result are consistent, and analyze the logic of the generated SQL statement to generate judgment results and judgment basis; If the result is consistent, the triple consisting of the question, SQL statement, and answer is stored in the success list; if the result is inconsistent, data correction is performed.

7. The method for automatically extracting query data from a heating ERP system using a large model according to claim 1, characterized in that, The process involves using business experts' feedback scores on candidate SQL statements, analyzing the differences between SQL statements with different scores through a large language model, summarizing generation experience, dynamically updating the expert experience base, and using the updated expert experience base to optimize the generation quality of subsequent SQL statements. Specifically, this includes: Collect issues from the heating ERP system, establish an initial issue list, and for each issue in the initial issue list, generate two or more candidate SQL statements as candidate groups based on the heating professional knowledge base and database description knowledge base through a large language model. Obtain the scores or directly labeled correct SQL statements from business experts regarding the candidate SQL statements; Based on the preset group retention strategy, candidate groups that contain both high-scoring and low-scoring SQL statements are selected. For the candidate group, the differences between high- and low-scoring SQL statements are analyzed using a large language model, and the experience in generating SQL for this problem is summarized. By using a large language model to determine the logical relationship between the summarized experience and the existing experience in the expert experience base, the expert experience base can be updated, deleted, or modified online to achieve online updates and enhancements.

8. The method for automatically extracting query data from a heating ERP system using a large model according to claim 1, characterized in that, The group retention strategy is as follows: a group is retained if both of the following conditions are met: at least one SQL statement in the candidate group has a score of 1 point and at least one SQL statement has a score of 0 points; or the SQL statements in the candidate group do not all have a score of 1 point and do not all have a score of 0 points.

9. An apparatus for automatically extracting query data from a heating ERP system using a large model, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to, when running the program instructions, execute the method as described in any one of claims 1-8 for automatically extracting query data from a heating ERP system using a large model.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a method for automatically extracting query data from a heating ERP system using a large model, as described in any one of claims 1-8 above.

Images