Method for generating an execution report of a large language model

Abstract

This application discloses a method for generating execution reports for a large language model, belonging to the field of large language model technology. The method includes: acquiring a user question and raw processing data input to a target large language model; generating an executable code fragment based on the user question, and executing the executable code fragment on the raw processing data to obtain execution data; encapsulating the executable code fragment and the execution data into an intermediate structure object; executing the intermediate structure object on the raw processing data, and obtaining a target template result based on the execution result; and generating a target execution report for the target large language model based on the user question, the target template result, and the raw processing data. This method significantly improves the accuracy and reliability of data analysis while maintaining the flexibility of natural language generation from the large language model.

Description

Technical Field

[0001] This application belongs to the field of large language model technology, and in particular relates to a method for generating execution reports of large language models. Background Technology

[0002] As enterprise data assets continue to expand, the need for business personnel to gain data insights through natural language queries is becoming increasingly urgent. However, existing natural language data analysis solutions generally suffer from two major drawbacks: illusion and lack of traceability. When generating answers, large language models often rely on statistical regularities in their pre-trained parameters rather than real data, resulting in numerical conclusions lacking underlying data support and making it impossible to verify the correctness of their calculation process. Even if some solutions introduce code interpreters to allow the model to generate executable code, they only reach a shallow verification level where the code can run, failing to identify hidden errors in the code that run but violate logical rules (such as violating structural constraints like monotonicity and arithmetic self-consistency). The final report conclusions may still have factual biases, making it difficult to meet the stringent data accuracy requirements of scenarios such as finance, government, and industry. Summary of the Invention

[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a method for generating execution reports for large language models, which significantly improves the accuracy and reliability of data analysis while maintaining the flexibility of natural language generation from large language models.

[0004] Firstly, this application provides a method for generating execution reports for large language models, including: Obtain user questions and raw processing data that are input into the target large language model; Based on the user's question, an executable code snippet is generated, and the executable code snippet is executed on the original processed data to obtain the executed data; The executable code fragment and the execution data are encapsulated into an intermediate structure object; The intermediate structure object is executed on the original processed data, and the target template result is obtained based on the execution result; Based on the user question, the target template result, and the original processing data, a target execution report for the target large language model is generated.

[0005] According to the execution report generation method of the large language model in this application, an intermediate structure object is established as evidence. Based on the user question, an executable code fragment is generated and executed on the original processed data to obtain execution data. An intermediate structure object is generated based on the executable code fragment and the execution data. The intermediate structure object is re-executed and verified on the original processed data to ensure the reproducibility of the conclusion values ​​and eliminate hidden errors of logical violations. In the final target execution report, each conclusion value is anchored to the corresponding row-level evidence and numerical key value, realizing full-link traceability and recalculation from user question to final report. While ensuring the flexibility of natural language generation of the large language model, it significantly improves the accuracy and reliability of data analysis.

[0006] According to one embodiment of this application, the step of generating an executable code fragment based on the user question and executing the executable code fragment on the original processing data to obtain execution data includes: Based on the raw processed data, construct the target text profile; Based on the target text profile, the user question is decomposed into multiple sub-questions carrying analytical intent; For each of the aforementioned sub-problems, a corresponding target calculation template is selected based on the analytical intent; Based on the target calculation template, the executable code snippet is generated, and the executable code snippet is executed on the original processed data to obtain insight values ​​and full row-level evidence as the execution data.

[0007] According to one embodiment of this application, constructing a target text profile based on the original processed data includes: The original processed data is parsed by splitting the table, and a column-level profile is independently constructed for each worktable. The column-level profile includes at least one of the following: column name, data type, semantic role, missing rate, number of unique values, extreme values ​​of numerical columns, mean of numerical columns, and frequency distribution of categorical columns. Based on each of the worksheets and the corresponding column-level profiles, the target text profile is constructed.

[0008] According to one embodiment of this application, the semantic roles include an identifier column, a dimension column, and a measure column; the step of independently constructing column-level profiles for each worksheet includes: If the column name of the first column of the worksheet is a preset identifier pattern, or if the first column is a numeric column and the proportion of unique values ​​reaches a first preset threshold, then the first column is determined to be the identifier column, and the first column can be any column in the worksheet. Alternatively, if the first column is an enumeration type or a time type, or if the first column is a text column and the number of unique values ​​is less than a second preset threshold, the first column is determined to be the dimension column; Alternatively, the numeric column that is not an identifier can be identified as the measure column.

[0009] According to one embodiment of this application, the step of decomposing the user question into multiple sub-questions carrying analytical intent based on the target text profile includes: Based on the target text profile, matching is performed in the preset intent menu to decompose the user question into multiple sub-questions carrying the analysis intent, wherein the preset intent menu is set in layers according to intent priority.

[0010] According to one embodiment of this application, the step of executing the intermediate structure object on the original processed data and obtaining the target template result based on the execution result includes: The code snippet in the intermediate structure object is executed on the original processed data to obtain recalculated data as the execution result; If the error between the recalculated data and the execution data in the intermediate structure object meets the preset tolerance condition, the target template result is obtained based on the recalculated data.

[0011] According to one embodiment of this application, when the error between the recalculated data and the execution data in the intermediate structure object meets a preset tolerance condition, obtaining the target template result based on the recalculated data includes: If the error between the recalculated data and the execution data in the intermediate structural object meets the preset tolerance condition, the corresponding structural self-consistency constraint is determined based on the analysis intent category of the intermediate structural object. Given that the intermediate structural object satisfies the structural self-consistency constraint, the target template result is obtained based on the recalculated data.

[0012] According to one embodiment of this application, the structural self-consistency constraint includes at least one of the following: In the intermediate structure object, row-level data is arranged monotonically and non-increasingly according to the metric value; The proportion of each row of data in the intermediate structure object is equal to the ratio of its own metric value to the total amount of data, and the derived quantity is equal to the result of independent recalculation of the row-level data. The dimension values ​​of the row-level data in the intermediate structure object exist in the corresponding dimension columns of the original processed data; When the original processed data is not empty, the row-level data in the intermediate structure object is not empty and not all of the measures are missing.

[0013] According to one embodiment of this application, generating the target execution report of the target large language model based on the user question, the target template result, and the original processing data includes: Under preset synthesis constraints, a target execution report of the target large language model is generated based on the user question, the target template result, and the original processing data. The preset synthesis constraints include the target large language model extracting numerical values ​​from the target template results as the numerical source of the target execution report; and / or, in the case that no verified numerical values ​​exist in the target template results, the target execution report adopts a qualitative description or omits the corresponding content.

[0014] According to one embodiment of this application, after generating the target execution report of the target large language model based on the user question, the target template result, and the original processing data, the method further includes: Extract the report numbers from the target execution report to form a report number set; The difference between the reported set of numbers and the target template result is used to obtain the unmapped numbers; Based on the unmapped number, exception handling is performed on the target execution report, and the exception handling includes at least one of the following: An error message will be generated for the unmapped number; In the target execution report, the content corresponding to the unmapped numbers is marked as abnormal; Based on the target template results, the content corresponding to the unmapped numbers is corrected or removed.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the method for generating execution reports of a large language model provided in this application embodiment; Figure 2 This is the second flowchart illustrating the method for generating execution reports for a large language model provided in this application embodiment; Figure 3 This is the third flowchart illustrating the method for generating execution reports for a large language model provided in this application embodiment; Figure 4 This is a schematic diagram of the large model prompt words for problem decomposition provided in the embodiments of this application; Figure 5 This is a schematic diagram of the common header of the calculation template provided in the embodiments of this application; Figure 6 This is a schematic diagram of the operator body of the calculation template provided in the embodiments of this application; Figure 7 This is a schematic diagram of the row-level evidence provided in the embodiments of this application; Figure 8 This is a schematic diagram of the numerical key value provided in the embodiments of this application; Figure 9 This is a schematic diagram of the insight mining and verification process provided in the embodiments of this application; Figure 10 This is a schematic diagram of the large model prompt words for report generation provided in the embodiments of this application; Figure 11 This is a flowchart illustrating the numerical auditing process provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] The following description, in conjunction with the accompanying drawings, details the method for generating execution reports of a large language model, the electronic device, and the readable storage medium provided in this application, through specific embodiments and application scenarios.

[0020] The method for generating execution reports for large language models can be applied to terminals, specifically executed by hardware or software within the terminal.

[0021] The execution report generation method for a large language model provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the execution report generation method for the large language model. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The execution report generation method for a large language model provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0022] like Figure 1 As shown, the method for generating the execution report of this large language model includes steps 110-150.

[0023] Step 110: Obtain the user questions and raw processing data input into the target large language model.

[0024] Among them, user questions are natural language analysis requirements or query instructions that users input into the target large language model.

[0025] The raw data used as the initial data source for analysis can be a spreadsheet file containing one or more worksheets.

[0026] In this embodiment, a natural language analysis command input by the user can be received as a user question through a human-computer interaction interface, and a raw table file containing one or more worksheets can be loaded from local storage or an external data source as raw processing data through a data reading interface. The user question provides the analysis intent and target constraints for the raw processing data, and the raw processing data provides the underlying factual basis and calculation object for responding to the user question.

[0027] Step 120: Based on the user's question, generate an executable code snippet and execute the executable code snippet on the original processing data to obtain the executed data.

[0028] Among them, the executable code snippets are data processing scripts or query instructions generated based on user questions and capable of running directly on the raw processed data.

[0029] Execution data refers to the structured computation results or intermediate processing products output after an executable code fragment is run on the original processed data.

[0030] In this step, executable code snippets can be generated based on the user's analytical intent and the structural characteristics of the data to filter, aggregate, or perform statistical operations on the raw data.

[0031] The executable code snippet is applied to the original processing data for parsing and computation, thereby outputting execution data containing structured computation results or intermediate processing products.

[0032] Step 130: Encapsulate the executable code fragment and execution data into an intermediate structure object.

[0033] The intermediate structure object is an intermediate evidence carrier built based on executable code fragments and execution data, containing reusable data processing logic and structured result features. It is used to encapsulate the correspondence between analysis intent and data output, providing traceable and verifiable calculation basis for the subsequent generation of target execution reports.

[0034] In this embodiment, the executable code fragment and execution data can be encapsulated to obtain an intermediate structure object.

[0035] Step 140: Execute the intermediate structure object on the original processed data, and obtain the target template result based on the execution result.

[0036] The execution result is the recalculated values ​​and row-level evidence produced by rerunning the executable code fragments encapsulated in the intermediate structure object on the original processed data.

[0037] The target template result is the verified insight result that is retained as the basis for report generation after the intermediate structure object is re-executed on the original processed data and the execution result is verified.

[0038] In this step, the executable code fragment encapsulated in the intermediate structure object is re-run on the original processed data to obtain recalculated values ​​and row-level evidence as execution results. The execution results are verified, and the verification is judged based on the difference between the execution results and the execution data in the intermediate structure object. The target template result is obtained based on the intermediate structure object after verification.

[0039] Step 150: Based on the user question, target template results, and raw processing data, generate a target execution report for the target large language model.

[0040] The target execution report is a final output report generated by the target large language model after completing the user question analysis, which integrates the user question, target template results, and raw processing data. It is used to clearly present the analysis conclusions, calculation basis, and traceable data evidence.

[0041] In this embodiment, the analytical intent expressed by the user's question can be used as the main thread of the report, the verified insights provided by the target template results can be used as the core evidence, and the original processed data can be used as the traceable source of facts. The three are comprehensively organized and expressed in natural language to finally generate a target execution report that not only responds to user needs but also has calculation basis and data support.

[0042] In this embodiment, by transforming the statistical inference of the target large language model into deterministic code operations and binding the analytical intent, computational logic, and underlying data support into a structured intermediate structure object, the technical defects of numerical illusion and untraceable conclusions are solved. Furthermore, by re-executing and verifying the intermediate structure object on the original processed data, the reproducibility of the conclusion values ​​is ensured and hidden errors of logical violations are eliminated. In the final target execution report, each conclusion value is anchored to the corresponding row-level evidence and numerical key value, realizing full-link traceability and recalculation from user questions to the final report. While ensuring the flexibility of natural language generation of the large language model, the accuracy and reliability of data analysis are significantly improved.

[0043] According to the execution report generation method of the large language model provided in this application, an intermediate structure object is established as evidence. Based on the user question, an executable code fragment is generated and executed on the original processed data to obtain execution data. An intermediate structure object is generated based on the executable code fragment and the execution data. The intermediate structure object is re-executed and verified on the original processed data to ensure the reproducibility of the conclusion values ​​and eliminate hidden errors of logical violations. In the final target execution report, each conclusion value is anchored to the corresponding row-level evidence and numerical key value, realizing full-link traceability and recalculation from the user question to the final report. While ensuring the flexibility of natural language generation of the large language model, the accuracy and reliability of data analysis are significantly improved.

[0044] In some embodiments, based on a user question, an executable code snippet is generated and executed on the original processed data to obtain execution data, including: Based on the raw processed data, construct a target text profile; Based on the target text profile, user questions are broken down into multiple sub-questions carrying analytical intent; For each sub-problem, select the corresponding target calculation template based on the analysis intent; Based on the target calculation template, an executable code snippet is generated and executed on the original processed data to obtain insight values ​​and full row-level evidence as execution data.

[0045] Among them, the target text profile is a descriptive object formed by abstracting and representing the fields, structure, and statistical features of the original processed data.

[0046] The analytical intent is the type of deterministic data processing logic (such as aggregation, filtering, sorting, comparison, etc.) that the sub-problem points to.

[0047] A subproblem is the smallest analytical unit that can be independently mapped to the target computation template, obtained by breaking down the user problem according to the analysis intent.

[0048] The target computation template is a predefined mapping configuration object between analytical intent and deterministic computation logic, used to transform the analytical intent pointed to by the subproblem into an executable code snippet that can run on the original processed data.

[0049] Insight values ​​are the aggregated results or statistical conclusions produced after an executable code snippet is executed on the raw processed data, serving as conclusive data support for the generation of subsequent reports.

[0050] Full row-level evidence is the collection of raw data rows that an executable code snippet depends on or produces during execution, used to provide underlying data tracing and traceability verification for insight values.

[0051] In this embodiment, the fields, structure, and statistical features of the raw processed data are abstracted and represented to construct a target text profile. Based on the target text profile, the user's natural language question is decomposed into multiple sub-questions carrying clear analytical intentions. For the analytical intention pointed to by each sub-question, a corresponding predefined target calculation template is matched. Then, the target calculation template is transformed into an executable code fragment that can run on the raw processed data and executed, producing insight values ​​as conclusive data support and full row-level evidence for underlying data tracing and traceability verification, which together constitute the execution data.

[0052] In some embodiments, a target text profile is constructed based on the raw processed data, including: The original processed data is split and parsed, and a column-level profile is independently built for each worksheet. The column-level profile includes at least one of the following: column name, data type, semantic role, missing rate, number of unique values, extreme values ​​of numerical columns, mean of numerical columns, and frequency distribution of categorical columns. Based on each worksheet and its corresponding column-level profile, construct the target text profile.

[0053] Among them, column-level profiling is a collection of column-level metadata extracted from a single column of a worksheet, based on dimensions such as field definition, data quality, and statistical characteristics.

[0054] Column name is the field identifier of a column in a worksheet; data type is the format category of the data in the column (such as numeric, text, date, etc.); semantic role is the meaning of the column in the business context (such as "order amount", "user ID", "order time", etc.); missing rate is the proportion of missing values ​​in the column to the total number of rows; number of unique values ​​is the number of unique values ​​in the column; extreme values ​​of numeric columns are the maximum and minimum values ​​in the column; mean of numeric columns is the arithmetic mean of all numeric records in the column; frequency distribution of categorical columns is the statistical count of the number of times each category value appears in the column.

[0055] In this embodiment, the original processed data is parsed in separate tables, and column-level profiles are extracted independently for each worktable, covering at least one of the following: column name, data type, semantic role, missing rate, number of unique values, extreme values ​​of numerical columns, mean of numerical columns, and frequency distribution of categorical columns.

[0056] Based on each worksheet and its corresponding column-level profiles, an abstract description of the overall structure and characteristics of the original processed data, namely the target text profile, is aggregated and constructed, providing a structured data cognitive foundation for subsequent user question intent decomposition and calculation template matching.

[0057] In some embodiments, semantic roles include identifier columns, dimension columns, and measure columns; column-level profiles are built independently for each worksheet, including: If the first column of a worksheet is named in a preset identifier pattern, or if the first column is a numeric column and the percentage of unique values ​​reaches a first preset threshold, then the first column is determined to be an identifier column, and the first column can be any column in the worksheet. Alternatively, if the first column is an enumeration type or a time type, or if the first column is a text column and the number of unique values ​​is less than the second preset threshold, the first column is determined to be a dimension column; Alternatively, define the numeric column that is not an identifier as a measure column.

[0058] The identifier column is used to uniquely identify data rows, such as order ID, user number, etc.

[0059] Dimension columns are used for grouping, filtering, or classifying data, such as region, date, product type, etc.

[0060] The measure column is a numeric column other than the identifier column, used to carry quantitative indicators that can be aggregated and calculated, such as sales, quantity, and duration.

[0061] The default identifier pattern is a set of predefined regular expressions or keywords used to match whether column names conform to common primary key naming rules.

[0062] The first preset threshold is to determine whether the proportion of unique values ​​in a column to the total number of rows in that column is high enough to be considered as the lower limit of the value of the identifier column.

[0063] The second preset threshold is the upper limit of the values ​​for determining whether the number of unique values ​​in a text column is small enough to be considered as an enumerated dimension column.

[0064] In this embodiment, for the first column, it is determined whether it is an identifier column used to uniquely identify data rows by whether the column name matches a preset identifier pattern, or whether the column is a numeric column and the proportion of unique values ​​reaches a first preset threshold.

[0065] Whether a column is a dimension column used for data grouping and classification is determined based on whether the first column is an enumeration type, a time type, or a text column with fewer than a second preset threshold number of unique values.

[0066] The remaining numerical columns that were not identified as identifier columns are identified as metric columns that can be aggregated and computed into quantitative indicators. This enables the automatic classification and labeling of the semantic roles of each column in the original processed data, providing a structured semantic foundation for the subsequent decomposition of sub-problems and accurate matching of target calculation templates.

[0067] In some embodiments, based on the target text profile, the user question is decomposed into multiple sub-questions carrying analytical intent, including: Based on the target text profile, matching is performed in the preset intent menu to decompose the user's question into multiple sub-questions carrying analysis intents. The preset intent menu is set in layers according to intent priority.

[0068] The preset intent menu is a set of data analysis intent options predefined and organized in a hierarchical manner according to the target text profile. It is used to map and break down the user's natural language questions into multiple sub-questions carrying clear analysis intents.

[0069] In this embodiment, intent matching is performed on a preset intent menu based on the target text profile. By matching the user's natural language question with intent options of different priorities in the menu, the user's question is decomposed into multiple sub-questions carrying clear analytical intents. This achieves a structured decomposition of complex user questions, laying the foundation for matching target calculation templates for each sub-question and generating executable code snippets.

[0070] It should be noted that the data analysis intent options in the preset intent menu can be divided into multiple priority levels based on business commonality, computational complexity, or dependency. When matching user questions, higher priority intents will be matched first to ensure the rationality of the order of sub-question decomposition and the completeness of the analysis logic.

[0071] In some embodiments, an intermediate structure object is executed on the original processed data, and a target template result is obtained based on the execution result, including: Execute the code snippet in the intermediate structure object on the original processed data to obtain the recalculated data as the execution result; If the error between the recalculated data and the execution data in the intermediate structure object meets the preset tolerance conditions, the target template result is obtained based on the recalculated data.

[0072] Among them, the recalculated data is the calculation result obtained by re-executing the code snippet in the intermediate structure object on the original processed data, which is used to compare and verify with the execution data in the intermediate structure object.

[0073] Predefined tolerance conditions are predefined thresholds or judgment rules for determining whether the error between the recalculated data and the executed data is within an acceptable range.

[0074] In this embodiment, the code fragment encapsulated in the intermediate structure object is first re-executed on the original processed data to obtain recalculated data as the execution result. Then, the recalculated data is compared with the execution data pre-stored in the intermediate structure object for error.

[0075] If the error between the recalculated data and the execution data in the intermediate structure object meets the preset tolerance conditions, the validity and consistency of the execution result of the intermediate structure object are confirmed, and the recalculated data is directly used or further processed and output as the target template result. Thus, through the secondary verification of the template calculation logic, the accuracy and reliability of the output result are ensured.

[0076] In some embodiments, if the error between the recalculated data and the execution data in the intermediate structure object meets a preset tolerance condition, the target template result is obtained based on the recalculated data, including: If the error between the recalculated data and the execution data in the intermediate structural object meets the preset tolerance conditions, the corresponding structural self-consistency constraints are determined based on the analysis intent category of the intermediate structural object. Given that the intermediate structural object satisfies the structural self-consistency constraint, the target template result is obtained based on the recalculated data.

[0077] Among them, structural self-consistency constraints are a set of predefined rules corresponding to the analysis intent category of intermediate structural objects, used to verify whether the internal logical structure of intermediate structural objects is self-consistent. They may include constraints on the semantic role matching of columns involved in intermediate structural objects, the completeness of computational logic, and the standardization of output format.

[0078] In this embodiment, after the error between the recalculated data and the execution data pre-stored in the intermediate structure object meets the preset tolerance conditions and the preliminary verification at the numerical level is completed, the corresponding structural self-consistency constraints are further determined according to the analysis intent category to which the intermediate structure object belongs, such as trend analysis, anomaly detection, attribution analysis, etc., and the internal logical structure of the intermediate structure object is verified.

[0079] Verification can include checking whether the columns referenced in the intermediate structure object match the semantic roles in the target text profile, whether the calculation logic fully covers the analysis steps required for the intent category, and whether the output format conforms to the specifications.

[0080] When the intermediate structural object passes both numerical error verification and structural self-consistency constraint verification, the final target template result is determined based on the recalculated data. This cross-validation of numerical and structural dimensions ensures that the output result is not only numerically accurate but also consistent with the preset intent in terms of analytical logic, thus avoiding analytical deviations caused by template structural defects or semantic role mismatch.

[0081] In some embodiments, structural self-consistency constraints include at least one of the following: In intermediate structure objects, row-level data is arranged monotonically and non-increasingly according to the measure value; In the intermediate structure object, the proportion of each row of data is equal to the ratio of its own metric value to the total amount of data, and the derived quantity is equal to the result of independent recalculation of the row-level data; The dimension values ​​of row-level data in the intermediate structure object exist in the corresponding dimension columns of the original processed data; When the original processed data is not empty, the row-level data in the intermediate structure object is not empty and not all of the measures are missing.

[0082] In this embodiment, the row-level data in the intermediate structure object is arranged monotonically and non-increasingly according to the metric value to ensure the sorting standard of analysis results such as ranking.

[0083] Verify whether the proportion of each row's data is equal to the ratio of its metric value to the total data volume, and whether derived quantities (such as cumulative values, average values, etc.) can be independently recalculated from row-level data, so as to ensure the logical closed loop of aggregation calculation and derived indicators.

[0084] Verify that all dimension values ​​of the row-level data in the intermediate structure object exist in the corresponding dimension columns of the original processed data to prevent the appearance of phantom dimension values ​​that do not exist in the original data.

[0085] Provided that the original processed data is not empty, the row-level data of the intermediate structure objects must not be empty and all measures must not be missing, so as to ensure the validity and completeness of the analysis results. Thus, by verifying at least one of the above constraints, the reliability of the template results at the structural level can be further confirmed.

[0086] In some embodiments, a target execution report for the target large language model is generated based on the user question, the target template result, and the raw processing data, including: Under preset synthesis constraints, a target execution report of the target large language model is generated based on the user question, target template results, and raw processing data. Among them, the preset synthesis constraints include the target large language model extracting numerical values ​​from the target template results as the numerical source of the target execution report; and / or, in the case that there are no verified numerical values ​​in the target template results, the target execution report adopts a qualitative description or omits the corresponding content.

[0087] In this embodiment, when generating the target execution report, the report content is organized by integrating user questions, target template results, and raw processing data within the framework of preset synthesis constraints. The large language model is required to extract numerical values ​​from the target template results that have undergone double verification (numerical error verification and structural self-consistency verification) as the numerical basis for the report, ensuring that every piece of data can be traced back to the verified calculation results. When there are no corresponding verified numerical values ​​in the target template results, the constraint model uses qualitative descriptions (such as "rising trend" or "highest proportion") to describe the analysis conclusions, or directly omits this part of the content, thereby ensuring the readability of the report while preventing misleading statements due to lack of data support.

[0088] In some embodiments, after generating the target execution report of the target large language model based on the user question, the target template result, and the raw processing data, the method for generating the execution report of the large language model further includes: Extract the reported figures from the target execution report to form a set of reported figures; The difference between the reported set of numbers and the target template result is used to obtain the unmapped numbers; Exception handling is performed on the target execution report based on unmapped numbers. The exception handling includes at least one of the following: Generate an error message for unmapped numbers; In the target execution report, mark the content corresponding to unmapped numbers as anomalies; Based on the target template results, the content corresponding to the unmapped numbers is corrected or removed.

[0089] The reported figures are numerical values ​​extracted from the text content of the target execution report.

[0090] The report set of numbers is a collection of all numerical values ​​extracted from the text content of the target execution report. It serves as the audit basis for subsequent difference comparison with the target template results to identify unmapped numbers.

[0091] Unmapped numbers are values ​​obtained by performing a difference operation between the set of numbers in this report and the target template result, and for which no corresponding source can be found in the target template result.

[0092] In this embodiment, all numerical values ​​are extracted from the text content of the generated target execution report to form a report number set. The difference operation is performed between this report number set and the target template result to obtain unmapped numbers whose corresponding sources cannot be found in the target template result. Based on these unmapped numbers, the target execution report is numerically audited. For example, values ​​in the target execution report that may be generated by model illusion or are not verified are identified and marked, thereby triggering the correction or regeneration of the target execution report. This adds a post-hoc defense after the target execution report is generated, ensuring that all values ​​in the target execution report finally output to the user are verifiable and traceable to the verified template result, further improving the credibility and accuracy of the report.

[0093] Anomaly handling of target execution reports based on unmapped numbers may include: generating anomaly prompts to inform of audit results, marking the content corresponding to unmapped numbers in the target execution report as anomalies to indicate risks, or correcting or removing relevant content based on the target template results to ensure that the values ​​in the target execution report are all derived from verified evidence, thereby forming a closed-loop process from numerical auditing to report quality control.

[0094] In related technologies, large-scale model data reporting solutions are prone to data illusions when processing complex tables due to a lack of reliable numerical traceability and verification mechanisms. Specifically, they suffer from three major drawbacks: First, traceability and verification are weak. The direct generation paradigm of large models cannot perform data traceability; while the code generation paradigm produces outputs, it lacks row-level evidence and struggles to detect hidden logical errors. Second, verification mechanisms are insufficient. Current verification is mostly based on single values, lacking comprehensive numerical auditing and formal constraints after report generation. Third, the computation is not reproducible. The analysis intent is not mapped to deterministic computation, resulting in an uncontrollable and unreproducible data retrieval process.

[0095] In this embodiment, an insight card, or intermediate structure object, is inserted between analysis and generation as an intermediate evidence layer. This card contains executable code and full line-level evidence. Data is retrieved by mapping the analysis intent to a deterministic calculation template, and code verification, semantic self-consistency verification, and full numerical audit of the synthesized report are performed sequentially to ensure that all values ​​in the target execution report are traceable and verifiable.

[0096] The following describes a specific implementation of a method for generating execution reports for a large language model.

[0097] Between table analysis and report generation, an insight card containing executable code and full row-level evidence is constructed as a structured intermediate evidence layer. Deterministic calculation templates driven by analytical intent retrieve data from the original table. The reproducibility of the data is verified by re-executing the code, and then a second verification is performed by combining semantic self-consistency verification unrelated to the code logic. Finally, after the report is synthesized, post-event numerical audit is implemented by formal constraints that the report's numerical set is included in the verified evidence set, forming a complete closed loop of deterministic data retrieval—code replay and semantic self-consistency verification—evidence constraint synthesis—post-event numerical audit.

[0098] like Figure 2 As shown, the system receives user questions and standardizes the corresponding data tables into two formats: a DataFrame for code execution and Markdown as the baseline facts for subsequent steps. It then deconstructs the intent behind the user questions, using pre-defined code templates to extract insights and obtain insight cards. The values ​​on the insight cards are re-validated, sorted based on priority, and the top _k insights are selected. Finally, the user questions, data tables, and insight cards are integrated using a large model to generate a data report.

[0099] Specifically, this may include: Step 1, data understanding and table profiling construction, where the raw data includes tables, the tables are parsed and processed independently by worksheet, the original data tables are retained for deterministic calculations, and text profiling is constructed for use by large models.

[0100] Step 2: Analyze intent-driven problem decomposition. The large language model decomposes the user's problem into multiple sub-problems carrying the analysis intent.

[0101] Step 3: Deterministic calculation template mining. Select target calculation templates according to the sub-problem analysis intent and generate executable code snippets to produce insight values ​​and row-level evidence, which are then encapsulated as insight cards, i.e., intermediate structure objects.

[0102] Step 4: Code recalculation and semantic self-consistency verification and sorting. For each card, first re-execute the code to verify reproducibility, then use semantic self-consistency verification to identify structural self-consistency errors, and sort the verified cards to obtain the subset of verified cards.

[0103] Step 5: Report generation and post-audit of evidence-bound data. The report is synthesized using only verified cards as numerical evidence, and the difference between the reported figures and the evidence is calculated to mark unsubstantiated figures.

[0104] In this embodiment, in step one, as follows Figure 3As shown, the table files are read and processed independently by worksheet, retaining the original data tables for deterministic calculations while constructing a compact target text profile for large language models to read. Semantic roles are inferred for each column: if the column name matches an identifier pattern such as "serial number / number / encoding," or if the percentage of unique values ​​in a numerical column exceeds a threshold, it is classified as an identifier column; if it is an enumeration or time type, or if the number of unique values ​​in a text column is limited, it is classified as a dimension column; other non-identifier numerical columns are classified as measure columns. The missing rate, number of unique values, extreme values ​​and mean values ​​of numerical columns, and frequency of categorical columns are summarized to form a column profile. In multi-table scenarios, multiple independent data tables are generated by worksheet and merged into a single master table by file dimension, with a new data source column added to distinguish the source. Finally, the first few rows of each table are extracted as a preview of the target text profile.

[0105] In step two, the user question, data description, column profile, and row preview are fed into the large language model, which decomposes them into multiple sub-questions carrying analytical intent. The large model's prompts for question decomposition are as follows: Figure 4 As shown, each sub-question includes a sub-question identifier, a natural language question, an analytical intent, a priority, indicator specifications, suggested grouping dimensions, and a source table identifier. The analytical intent is selected from a preset intent menu and hierarchically prioritized, ensuring that the decomposition results simultaneously cover analytical needs at different levels, such as global scale, entity ranking, comparison, and mechanism inference. The hierarchy and constraints of the intent menu are shown in Table 1.

[0106] Table 1

[0107] In step three, for each sub-problem, the metric column and grouping column are determined based on the indicator specifications and grouping dimensions, and the aggregation method is determined according to the semantic role of the metric column. The aggregation method is : ,like For identifier columns; ,like For dimension columns; ,like This is a measure column.

[0108] Subsequently, based on the analytical intent, a deterministic target calculation template (including ranking and percentage, total, ratio, two-entity comparison, cost-driven, mechanism inference, etc.) is selected. An executable code snippet is dynamically generated to extract data from the original processed data, producing insight values ​​and full row-level evidence, which are then encapsulated into insight cards, i.e., intermediate structure objects. Each card carries: card identifier, natural language description, analytical intent, numerical value, row-level evidence, the executable code snippet that generated the card, and source table identifier. Derived evidence for report reference is then derived from the row-level evidence and numerical values. The numerical values ​​are strictly taken from a subset of the card's numerical value and the row-level evidence, and subset verification confirms that each numerical value in the derived evidence can be mapped back to the card.

[0109] The deterministic calculation template consists of a common header declaring the data source and variables, and an operator body encapsulating the aggregation logic. The system automatically matches the data source based on the sub-problem and parses the measure column, grouping column, and aggregation method. Then, based on the analysis intent (such as ranking, total, ratio, etc.), it selects the corresponding operator from the template library, fills in the parameters, and generates an executable code snippet. If the code fails to run, it is automatically skipped. For example, taking a sales details table, if a user asks "What are the top 3 regions in terms of sales volume?", the system parses the analysis intent as ranking, the measure column as "sales volume", the grouping column as "region", and the truncation value as 3. The dynamically generated code snippet consists of a common header and an operator body. Public head, such as Figure 5 As shown, it is responsible for declaring the data source and variables; the operator body, such as Figure 6 As shown, fixed aggregation and sorting logic is encapsulated; ultimately, row-level evidence is obtained, such as... Figure 7 As shown; numeric key-value pairs, such as Figure 8 As shown.

[0110] In step four, two layers of verification are performed on each card. The first layer is code re-execution reproducibility verification: the code snippet is re-executed on the original table, and the resulting numerical key value (execution result) is compared with the row-level evidence one by one with the card records, using a relative error tolerance. Judgment, that is For recording values, When the value is to be re-executed:

[0111] Any missing or excessive key value, inconsistent row numbers or fields in row-level evidence, or abnormal code execution will result in disqualification. The results of this layer of confirmation cards can be reliably reproduced using the attached code.

[0112] The second layer performs semantic self-consistency verification: This layer of verification is performed by an independent verifier. The verifier takes cards, row-level evidence, numerical keys and source tables as inputs and applies a set of structural self-consistency constraints according to the card type. If any constraint is violated, the card is judged as failing.

[0113] The specific verification rules are as follows, categorized by card type: Ranking, percentage, structure, attribution, and tail extrema categories: Monotonicity constraint: Requires that row-level evidence must be arranged monotonically and non-increasingly according to the metric value.

[0114] Arithmetic self-consistency constraint: requires the percentage of each row. Strictly equal to the measure value of that row With total amount ratio Furthermore, derived values ​​in numeric keys (such as the first value, the first digit, etc.) The total of items must be equal to the result of independent recalculation of the row-level evidence.

[0115] Dimension Existence Constraint: For categories with grouping labels, the dimension values ​​in the row-level evidence must actually exist in the corresponding dimension columns of the source table.

[0116] Total Amount Category: Degradation detection constraint: When the source table is not empty, row-level evidence must not be empty, and all measures must not be missing.

[0117] Taking "top 3 in sales volume in each region" as an example, the semantic self-consistency validator will perform the following checks: Are the sales figures of the top three industry-level evidence strictly monotonically non-increasing? Does the percentage of each sector equal the ratio of that sector's sales to its total sales? Does the sum of the first and third values ​​in a numerical key equal the result of independent recalculation based on row-level evidence? Check whether the region labels actually exist in the "Region" column of the source table.

[0118] If the card mistakenly uses the "cost" column as the "sales revenue" column during generation, its code can still run normally and pass the first level of validation. However, in the second level of validation, because the consistency of the proportion of actual sales revenue or the existence invariant of the dimension is violated, the card will be accurately identified as a hidden error and rejected.

[0119] Cards validated through two layers are sorted and deduplicated using a multi-level key: a three-level sorting key is used, prioritizing validated cards, then analyzing intent, and finally descending the confidence level. Deduplication is further performed by category and sub-question, with a coverage constraint ensuring at least one card is retained for each sub-question. This process ultimately selects the most concise and optimized cards. A subset of verified cards .

[0120] like Figure 9 As shown, the parsing metrics and grouping columns: read the fields corresponding to the user's question, and identify the metrics and grouping dimensions that need to be calculated.

[0121] Determine semantic roles (branching decisions) and divide fields into three categories: Identifiers: use Count to count; Dimension columns: use Numeric to remove duplicates; Measure columns: use Sum to sum.

[0122] Match the calculation template and generate code. Based on the semantic role of the field, call the preset calculation template and automatically generate executable code.

[0123] Encapsulate insight cards; package the code execution results into insight cards.

[0124] Verification 1: Code replay, failure: remove this insight directly and record the exception information; pass: proceed to the next level of verification; Verification 2: Semantic consistency, failure: remove this insight and record the exception information; pass: generate derived evidence.

[0125] Confidence ranking and simplified output: All legitimate insights are ranked according to confidence level, simplified, and then output for subsequent report generation.

[0126] In step five, as Figure 10 As shown, the user question, data description, data table preview, and verified card set are displayed. The data is fed into a large model to synthesize an article-level report. Evidence constraints are imposed during synthesis: all figures in the text must come from [the relevant source]. Verified numerical values ​​and derived evidence are included; data table previews are for understanding dimensions and column names only, and numbers should not be extracted or inferred from them; when a fact is not supported by verified numerical values, it should be expressed qualitatively or omitted, and guessing numerical values ​​is prohibited.

[0127] After the report is generated, all numbers are extracted from the main text to form the report's data set. Allowed sets of numbers Depend on The set is the union of the numbers in the table and the pre-aggregated numbers from the entire table (number of rows, duplicate count, sum, mean, extreme values, percentage of top columns, etc.). Find the difference:

[0128] For those that cannot be mapped to The numbers are marked as unmapped, and a list of unmapped numbers is output to report the set of numbers. Included in the set of verified evidence Post-event verification, i.e.:

[0129] The generated report is evaluated using three complementary criteria in a multidimensional manner, and a report score is awarded. The arithmetic mean of the three:

[0130] Numerical accuracy criteria The self-consistency mechanism verifies whether the reported values ​​are consistent with the source table, eliminating the need for manual annotation of reference reports; information coverage criteria. Measuring the report's semantic coverage of key reference points; general quality criteria. The large model comprehensively scores the data based on dimensions such as reasoning depth, practicality, completeness, and coherence. The three criteria constrain the accuracy of the reported numbers, the completeness of key information, and the high quality of the overall expression. All of these criteria are automatically calculated without manual scoring and are consistent with the evidence constraints in the generation stage and the code re-execution verification.

[0131] like Figure 11 As shown, input the report text, use regular expressions to extract all numbers in the report; then process the numbers using format variants (integers, keeping 1 decimal place, keeping 2 decimal places) to obtain the report's set of numbers.

[0132] Construct allowed sets of numbers from two other data sources: verified cards: key_values / supporting_data; and pre-aggregated numbers across the entire table: number of rows / nunique / sum / mean / max / min / Top percentage.

[0133] Difference and inclusion relationship determination: Compare the reported set of numbers with the allowed set of numbers. Mapped numbers: allow; numbers that cannot be mapped in the difference set: mark as unmapped and count the number.

[0134] Output audit results: Output a list of unmapped numbers and their corresponding counts.

[0135] Branch judgment: Is the unmapped count zero? Yes: Validation passed, and all numbers in the report can be traced back to the verified evidence; No: Validation failed, and the report contains false values ​​that are not supported by original data.

[0136] The execution report generation method for large language models provided in this application is based on a deterministic calculation template of analysis intent. It directly maps the user's analysis intent to a predefined deterministic calculation template and executes fixed calculation logic on the original data table. This replaces the uncontrollable mode that relies on the large model to freely generate code, ensuring that the data retrieval and calculation are strictly reproducible.

[0137] A structured intermediate carrier containing code and line-level evidence, constructing insight cards containing executable code fragments and full line-level evidence, fully preserving computational logic and supporting data records, forming a recalculated chain of evidence.

[0138] The dual verification loop of code recalculation and semantic self-consistency verifies the reproducibility of card values ​​through code recalculation, and then applies semantic self-consistency verification to identify hidden errors in code that are runnable but logically violated; after the report is generated, a global numerical audit is performed to ensure that all reported values ​​are included in the set of verified evidence, thereby achieving a verification loop from local to global.

[0139] By combining intent-driven deterministic computation templates with dual verification, the accuracy and traceability of numerical values ​​in the table-to-report generation process are significantly improved. The deterministic template directly extracts data from the original data table, avoiding the irreproducibility and numerical illusions inherent in freely generated code from large models, making it suitable for large, complex, multi-worksheet scenarios. Code verification and semantic self-consistency dual-layer verification identify and eliminate hidden errors where the code is executable but the results violate structural self-consistency. Formal constraints that include the report's numerical set within the verified evidence set, along with post-project full-article auditing, construct a closed loop of pre-generation constraints and post-generation auditing. The generated insight cards contain executable code and row-level and derived dual-layer evidence, supporting independent auditing and recalculation, providing quantifiable and verifiable numerical compliance assurance for industrial-grade reports.

[0140] The execution report generation method for a large language model provided in this application can be executed by an execution report generation device for a large language model. This application uses the execution report generation device of the large language model executing the execution report generation method as an example to illustrate the execution report generation device for the large language model provided in this application.

[0141] This application also provides an apparatus for generating execution reports for a large language model.

[0142] The execution report generation device for this large language model includes: The acquisition module is used to acquire user questions and raw processing data that are input into the target large language model; The first processing module is used to generate executable code snippets based on user questions, and execute the executable code snippets on the original processing data to obtain the execution data; The second processing module is used to encapsulate executable code fragments and execution data into intermediate structure objects; The third processing module is used to execute intermediate structure objects on the original processed data and obtain the target template result based on the execution result; The fourth processing module is used to generate a target execution report for the target large language model based on the user's question, the target template results, and the raw processing data.

[0143] The execution report generation device for a large language model provided in this application establishes an intermediate structure object as evidence, generates an executable code fragment based on the user's question, executes the executable code fragment on the original processed data to obtain execution data, generates an intermediate structure object based on the executable code fragment and execution data, and re-executes and verifies the intermediate structure object on the original processed data to ensure the reproducibility of the conclusion values ​​and eliminate hidden errors of logical violations. In the final target execution report, each conclusion value is anchored to the corresponding row-level evidence and numerical key value, realizing full-link traceability and recalculation from the user's question to the final report. While ensuring the flexibility of natural language generation of the large language model, it significantly improves the accuracy and reliability of data analysis.

[0144] In some embodiments, the fourth processing module is used to construct a target text profile based on the original processed data; Based on the target text profile, user questions are broken down into multiple sub-questions carrying analytical intent; For each sub-problem, select the corresponding target calculation template based on the analysis intent; Based on the target calculation template, an executable code snippet is generated and executed on the original processed data to obtain insight values ​​and full row-level evidence as execution data.

[0145] In some embodiments, the fourth processing module is used to perform table partitioning and parsing on the original processed data, and to independently construct column-level profiles for each worksheet. The column-level profiles include at least one of column name, data type, semantic role, missing rate, number of unique values, extreme values ​​of numerical columns, mean of numerical columns, and frequency distribution of categorical columns. Based on each worksheet and its corresponding column-level profile, construct the target text profile.

[0146] In some embodiments, semantic roles include identifier columns, dimension columns, and measure columns; the fourth processing module is used to determine that the first column is an identifier column when the column name of the first column of the worksheet is a preset identifier pattern, or the first column is a numeric column and the proportion of unique values ​​reaches a first preset threshold, and the first column is any column in the worksheet. Alternatively, if the first column is an enumeration type or a time type, or if the first column is a text column and the number of unique values ​​is less than the second preset threshold, the first column is determined to be a dimension column; Alternatively, define the numeric column that is not an identifier as a measure column.

[0147] In some embodiments, the fourth processing module is used to match the target text profile in a preset intent menu and decompose the user question into multiple sub-questions carrying analysis intents, wherein the preset intent menu is set hierarchically according to intent priority.

[0148] In some embodiments, the third processing module is used to execute code snippets in an intermediate structure object on the original processed data to obtain recalculated data as the execution result; If the error between the recalculated data and the execution data in the intermediate structure object meets the preset tolerance conditions, the target template result is obtained based on the recalculated data.

[0149] In some embodiments, the third processing module is used to determine the corresponding structural self-consistency constraint based on the analysis intent category of the intermediate structural object when the error between the recalculated data and the execution data in the intermediate structural object meets the preset tolerance condition. Given that the intermediate structural object satisfies the structural self-consistency constraint, the target template result is obtained based on the recalculated data.

[0150] In some embodiments, structural self-consistency constraints include at least one of the following: In intermediate structure objects, row-level data is arranged monotonically and non-increasingly according to the measure value; In the intermediate structure object, the proportion of each row of data is equal to the ratio of its own metric value to the total amount of data, and the derived quantity is equal to the result of independent recalculation of the row-level data; The dimension values ​​of row-level data in the intermediate structure object exist in the corresponding dimension columns of the original processed data; When the original processed data is not empty, the row-level data in the intermediate structure object is not empty and not all of the measures are missing.

[0151] In some embodiments, the fourth processing module is configured to generate a target execution report of the target large language model based on the user question, the target template result, and the original processing data, under preset synthesis constraints. Among them, the preset synthesis constraints include the target large language model extracting numerical values ​​from the target template results as the numerical source of the target execution report; and / or, in the case that there are no verified numerical values ​​in the target template results, the target execution report adopts a qualitative description or omits the corresponding content.

[0152] In some embodiments, the fourth processing module is further configured to extract report numbers from the target execution report to form a report number set; The difference between the reported set of numbers and the target template result is used to obtain the unmapped numbers; Exception handling is performed on the target execution report based on unmapped numbers. The exception handling includes at least one of the following: Generate an error message for unmapped numbers; In the target execution report, mark the content corresponding to unmapped numbers as anomalies; Based on the target template results, the content corresponding to the unmapped numbers is corrected or removed.

[0153] The execution report generation device for the large language model in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific device.

[0154] The execution report generation device for the large language model in this embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this embodiment does not specifically limit the specific operating system.

[0155] The execution report generation device for the large language model provided in this application embodiment can achieve... Figures 1-3 , Figure 9 and Figure 11 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0156] In some embodiments, such as Figure 12 As shown, this application embodiment also provides an electronic device 1200, including a processor 1201, a memory 1202, and a computer program stored on the memory 1202 and executable on the processor 1201. When the program is executed by the processor 1201, it implements the various processes of the above-described large language model execution report generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0157] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0158] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described large language model execution report generation method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0159] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0160] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for generating execution reports of a large language model.

[0161] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0162] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described large language model execution report generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0163] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0164] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0167] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0168] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for generating execution reports for a large language model, characterized in that, include: Obtain user questions and raw processing data that are input into the target large language model; Based on the user's question, an executable code snippet is generated, and the executable code snippet is executed on the original processed data to obtain the executed data; The executable code fragment and the execution data are encapsulated into an intermediate structure object; The intermediate structure object is executed on the original processed data, and the target template result is obtained based on the execution result; Based on the user question, the target template result, and the original processing data, a target execution report for the target large language model is generated.

2. The method for generating execution reports for a large language model according to claim 1, characterized in that, Based on the user's question, an executable code fragment is generated, and the executable code fragment is executed on the original processed data to obtain execution data, including: Based on the raw processed data, construct the target text profile; Based on the target text profile, the user question is decomposed into multiple sub-questions carrying analytical intent; For each of the aforementioned sub-problems, a corresponding target calculation template is selected based on the analytical intent; Based on the target calculation template, the executable code snippet is generated, and the executable code snippet is executed on the original processed data to obtain insight values ​​and full row-level evidence as the execution data.

3. The method for generating execution reports for a large language model according to claim 2, characterized in that, The process of constructing a target text profile based on the original processed data includes: The original processed data is parsed by splitting the table, and a column-level profile is independently constructed for each worktable. The column-level profile includes at least one of the following: column name, data type, semantic role, missing rate, number of unique values, extreme values ​​of numerical columns, mean of numerical columns, and frequency distribution of categorical columns. Based on each of the worksheets and the corresponding column-level profiles, the target text profile is constructed.

4. The method for generating execution reports for a large language model according to claim 3, characterized in that, The semantic roles include an identifier column, a dimension column, and a measure column; The method of independently building column-level profiles for each worksheet includes: If the column name of the first column of the worksheet is a preset identifier pattern, or if the first column is a numeric column and the proportion of unique values ​​reaches a first preset threshold, then the first column is determined to be the identifier column, and the first column can be any column in the worksheet. Alternatively, if the first column is an enumeration type or a time type, or if the first column is a text column and the number of unique values ​​is less than a second preset threshold, the first column is determined to be the dimension column; Alternatively, the numeric column that is not an identifier can be identified as the measure column.

5. The method for generating execution reports for a large language model according to claim 2, characterized in that, Based on the target text profile, the user question is decomposed into multiple sub-questions carrying analytical intent, including: Based on the target text profile, matching is performed in the preset intent menu to decompose the user question into multiple sub-questions carrying the analysis intent, wherein the preset intent menu is set in layers according to intent priority.

6. The method for generating execution reports for a large language model according to claim 1, characterized in that, The step of executing the intermediate structure object on the original processed data and obtaining the target template result based on the execution result includes: The code snippet in the intermediate structure object is executed on the original processed data to obtain recalculated data as the execution result; If the error between the recalculated data and the execution data in the intermediate structure object meets the preset tolerance condition, the target template result is obtained based on the recalculated data.

7. The method for generating execution reports for a large language model according to claim 6, characterized in that, When the error between the recalculated data and the execution data in the intermediate structure object meets a preset tolerance condition, the target template result is obtained based on the recalculated data, including: If the error between the recalculated data and the execution data in the intermediate structural object meets the preset tolerance condition, the corresponding structural self-consistency constraint is determined based on the analysis intent category of the intermediate structural object. Given that the intermediate structural object satisfies the structural self-consistency constraint, the target template result is obtained based on the recalculated data.

8. The method for generating execution reports for a large language model according to claim 7, characterized in that, The structural self-consistency constraints include at least one of the following: In the intermediate structure object, row-level data is arranged monotonically and non-increasingly according to the metric value; The proportion of each row of data in the intermediate structure object is equal to the ratio of its own metric value to the total amount of data, and the derived quantity is equal to the result of independent recalculation of the row-level data. The dimension values ​​of the row-level data in the intermediate structure object exist in the corresponding dimension columns of the original processed data; When the original processed data is not empty, the row-level data in the intermediate structure object is not empty and not all of the measures are missing.

9. The method for generating execution reports for a large language model according to any one of claims 1-8, characterized in that, The process of generating a target execution report for the target large language model based on the user question, the target template result, and the original processing data includes: Under preset synthesis constraints, a target execution report of the target large language model is generated based on the user question, the target template result, and the original processing data. The preset synthesis constraints include the target large language model extracting numerical values ​​from the target template results as the numerical source of the target execution report; and / or, in the case that no verified numerical values ​​exist in the target template results, the target execution report adopts a qualitative description or omits the corresponding content.

10. The method for generating execution reports for a large language model according to any one of claims 1-8, characterized in that, After generating the target execution report of the target large language model based on the user question, the target template result, and the raw processing data, the method further includes: Extract the report numbers from the target execution report to form a report number set; The difference between the reported set of numbers and the target template result is used to obtain the unmapped numbers; Based on the unmapped number, exception handling is performed on the target execution report, and the exception handling includes at least one of the following: An error message will be generated for the unmapped number; In the target execution report, the content corresponding to the unmapped numbers is marked as abnormal; Based on the target template results, the content corresponding to the unmapped numbers is corrected or removed.

Images