An artificial intelligence natural language-based self-service data statistical analysis method and system

Abstract

This invention discloses a self-service data statistical analysis method and system based on artificial intelligence and natural language processing. It establishes a fixed mapping relationship between data source structure information and business terms by constructing and persistently storing a semantic mapping layer that is independent of the lifecycle of a single query. This allows for direct reading and reuse in subsequent natural language queries, solving the semantic drift problem. The semantic mapping layer is injected with rigid technical constraint configurations pre-defined in a structured configuration format. These constraints constrain a large language model to generate an imperative programming language analysis program capable of performing procedural tasks. After generation, the program undergoes mandatory verification according to the technical constraint configurations and runs in the execution environment, outputting structured results. Multi-turn interactions do not require the return of historical dialogues, ensuring consistent context. After 50 rounds of testing, this invention saves approximately 75% of input tokens compared to traditional solutions; after 20 repeated tests, the code generation and execution success rates both reached 100%.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence natural language processing technology, and in particular to a self-service data statistical analysis method and system based on artificial intelligence natural language, which is especially suitable for the natural language interface system in enterprise-level self-service business intelligence platforms. Background Technology

[0002] As the volume of business data continues to grow, business personnel have an increasingly urgent need for self-service data statistical analysis. Existing technical solutions have the following technical shortcomings:

[0003] 1. End-to-end architecture leads to semantic drift and resource waste. Current Natural Language to SQL (NL2SQL) and Natural Language BI technologies generally adopt an end-to-end architecture, where natural language queries are directly mapped to SQL statements. Under this architecture, field semantics need to be re-parsed for each query, lacking an independent, persistent semantic middleware layer. On the one hand, the same business term may be parsed as different fields in different query contexts, resulting in semantic drift; on the other hand, repeated semantic parsing leads to high costs associated with calling large language models.

[0004] 2. Limited Expressive Power of Declarative Query Languages. Existing technologies typically output declarative query languages ​​such as SQL. The expressive power of declarative languages ​​is limited by preset operators and aggregate functions, making it difficult to efficiently and concisely implement complex statistical analysis tasks involving iteration, conditional control flow, or dependence on external states, such as moving average calculations, conditional branching logic, and multi-step data transformations. Declarative languages ​​cannot directly call general-purpose programming language libraries (such as numerical computation and statistical modeling libraries). For tasks involving non-set operations or requiring procedural control, their expressive power and execution efficiency suffer from fundamental bottlenecks, failing to meet the needs of exploratory data analysis.

[0005] 3. Insufficient semantic mapping layer, exposing technical details. Existing natural language BI tools typically only support semantic mapping at the field level. Users still need to understand technical information such as data source names and table names, and cannot describe the complete data location using pure business language; the technical barrier has not been fundamentally eliminated.

[0006] 4. Context bloat in multi-turn interactions. In multi-turn statistical analysis interaction scenarios, traditional solutions typically require continuously feeding back the complete dialogue history to the large language model to maintain the dialogue state. This causes the context length to increase linearly with each interaction turn, token consumption to increase with each turn, and a large amount of irrelevant historical information may interfere with the large language model's understanding of the current request. Furthermore, in multi-turn dialogues, when the user's current request has a referential relationship with previous requests or when the request changes, existing solutions lack a systematic disambiguation mechanism to determine the execution request, leading to misunderstandings of the request.

[0007] 5. Weak adaptability to multiple data sources. Existing solutions mostly support only a single type of data source. For joint analysis across databases and tabular files, complex data warehouse federated query configurations are usually required, resulting in high system overhead and poor real-time performance.

[0008] 6. The quality of generated code is unstable and lacks determinism. Existing code generation schemes based on large language models may produce different output codes for the same input due to the inherent probabilistic sampling mechanism of the model. Furthermore, the constraints on the generated code in existing schemes are usually limited to persuasive suggestions at the prompt word level, lacking rigid constraints and mandatory verification at the system level, resulting in uncontrollable code quality.

[0009] 7. Code assets cannot be effectively preserved. Existing solutions require code to be regenerated for each query, and verified and usable code cannot be systematically saved and reused, which wastes computing resources and introduces unnecessary generation uncertainty risks.

[0010] Therefore, there is an urgent need for a self-service data statistical analysis method that can solve the above-mentioned technical problems. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and provide a self-service data statistical analysis method and system based on artificial intelligence natural language. By constructing a persistent semantic mapping layer and combining it with the rigid technical constraints configured in the system, it solves the technical problems in the prior art such as semantic drift, limited expressive ability, multi-turn interaction context expansion, unstable code generation quality, inability to effectively accumulate code assets, and reliance on professional technical personnel.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] like Figure 1 As shown, a self-service data statistical analysis method based on artificial intelligence and natural language processing includes the following steps:

[0014] Step 1: Connect to one or more data sources, including relational databases and / or tabular files;

[0015] Step 2: Construct and persist a semantic mapping layer that is independent of the lifecycle of a single natural language query. The semantic mapping layer establishes a fixed mapping relationship between data source structure information and business terms, so that it can be directly read and reused in subsequent natural language queries. The semantic mapping layer is used as an intermediate mapping layer between natural language statistical analysis requirements and executable data analysis programs.

[0016] Step 3: Receive the natural language statistical analysis request initiated by the user based on the semantic mapping layer;

[0017] Step 4: Inject the semantic mapping layer with a preset technical constraint configuration. Through the technical constraint configuration, the large language model is forced to generate an analysis program for an imperative programming language. The technical constraint configuration is preset in a structured configuration format. After the program is generated, the system automatically performs forced verification by parsing the generated code and constructing an abstract syntax tree based on the technical constraint configuration. Programs that do not conform to the constraints are rejected from entering the execution stage. This eliminates programs that do not conform to the preset technical constraints during the generation stage, ensuring the security and isolation of the subsequent execution environment.

[0018] Step 5: Run the validated analysis program in the execution environment and output the analysis results.

[0019] Furthermore, in step 5, the structured output of the analysis results embeds a clear requirement description returned by the large language model. The clear requirement description is a deterministic requirement that is satisfied by this code execution after processing by disambiguation rules.

[0020] Furthermore, the semantic mapping layer establishes a three-tiered mapping relationship between data source name, data table name, field name, and business terms; the semantic mapping layer persists storage in a structured format and supports version management, incremental updates, and version rollback; the data types of the fields include any one or more of numeric, text, date / time, boolean, and categorical types.

[0021] Furthermore, the semantic mapping layer is stored independently in the metadata database.

[0022] Furthermore, the technical constraint configuration described in step 4 is pre-configured in the system in the form of a structured configuration file and loaded into a rule object in memory when the system starts. When step 4 is executed, the system converts each constraint in the rule object into a corresponding generation instruction and verification rule.

[0023] Furthermore, the technical constraint configuration includes at least the following constraints: function signature constraints, used to limit the analysis program generated by the large language model to have preset standardized function entry and exit points, which are forcibly verified by the abstract syntax tree parsing and verification after generation; code generation strategy constraints, including preset code generation optimization instructions for different data source types, wherein, for relational data sources, the preset optimization instructions include at least: when grouping aggregation or multi-table join operations need to be performed, embedding and executing structured query language in the generated imperative program to process the initial data in the database, so as to reduce data transmission overhead and the loop processing cost of imperative language; output data structure constraints, embedding a preset structured output format definition, which is validated by the system after program execution, and the results that do not conform to the structured output format definition are intercepted and retried, wherein the structured output format definition includes preset fields for carrying the explicit requirement description; multi-turn dialogue requirement disambiguation rules, used to determine whether a requirement conflict occurs based on the main field set of the requirement output in a multi-turn dialogue scenario, and to determine the unique execution requirement.

[0024] Furthermore, the multi-turn dialogue requirement disambiguation rules include: the system parses the output main field sets corresponding to the current user requirement and the previous requirement, and determines whether a requirement conflict occurs by comparing the output main field sets; when multiple different output main field sets appear, the last output main field set is taken as the standard, and the constraints in the previous requirement that are compatible with the last output main field set are merged into the current execution requirement; the compatible constraints refer to the dimension defined by the constraint being semantically related to the data analysis dimension corresponding to the last output main field set; when an unavoidable semantic conflict is detected, the system directly returns failure, the reason for which is that the user's accurate meaning cannot be determined.

[0025] Furthermore, in the code generation strategy constraints, for relational data sources, the preset optimization instructions also include: requiring the generated structured query language to follow the rule of using aggregate functions to avoid loops, and the rule of using backticks to escape table names and field names.

[0026] Furthermore, the structured output format definition embedded in the output data structure constraint includes a variety of preset analysis result type definitions, each of which includes a visualization field for indicating the front-end visualization format; the technical constraint configuration also includes automatic inference rules from data features to the various analysis result types.

[0027] Furthermore, in step 4, the system combines the semantic mapping layer and the rule objects in memory through template concatenation, and converts them together into the input of the large language model. This allows the large language model to generate an imperative programming language analysis program under the dual constraints of the fixed semantic constraints provided by the semantic mapping layer and the program structure constraints provided by the rule objects.

[0028] Furthermore, when calling the large language model, the model temperature parameter is set to 0 and the random seed is fixed to ensure that the same input produces the same analysis program under the same inference environment.

[0029] Furthermore, after step 4 and before step 5, code verification and retry steps are included: the generated analysis program is subjected to syntax verification and dependency checks; the syntax verification uses an abstract syntax tree parsing method to check whether the code conforms to the basic syntax rules of the target programming language; the dependency check confirms that the modules, classes or functions referenced by the analysis program are available in the execution environment; if the verification fails, the error information is fed back to the large language model and regeneration is triggered, with a maximum of a preset number of retries.

[0030] Furthermore, the execution environment is configured with resource limits, function call blacklists, and network isolation policies; the execution environment is implemented based on container technology.

[0031] Furthermore, the construction and modification of the semantic mapping layer in step 2 includes: construction phase: the semantic mapping layer is initially constructed using any one or more of the following methods: automatic parsing based on the data source structure by the system, manual annotation by the user through the interface, definition by the user through natural language description, or automatic inference by the large language model based on the data source structure; modification and evolution phase: users are supported to supplement or modify the mapping results through natural language interaction, and the modified semantic mapping layer automatically updates the version number and retains the historical version.

[0032] Furthermore, when multiple data sources are connected, each data source establishes an independent connection, and the system manages the access permissions of each data source independently to ensure data isolation between data sources. When multiple data sources need to be correlated, the system automatically identifies the correlateable fields through the field mapping in the semantic mapping layer.

[0033] Furthermore, step 5 is followed by a code persistence and reuse step: the system extracts the explicit requirement description returned by the large language model from the structured output of the analysis results, establishes an association between the requirement description and the corresponding analysis program, and persists it; when the system receives a new natural language statistical analysis requirement, it calculates the similarity between the new requirement and the persisted requirement description using the embedded vector embedding cosine similarity algorithm. If the similarity is greater than a preset threshold, it is determined to be the same requirement, and the associated analysis program is directly called for execution without having to call the large language model again to regenerate the code; the preset threshold is a configurable parameter between 0.85 and 0.95.

[0034] A self-service data statistical analysis system based on artificial intelligence and natural language processing includes:

[0035] The data access module is used to access one or more data sources;

[0036] The semantic mapping management module is used to build and persist the semantic mapping layer that is independent of the lifecycle of a single natural language query, and establish a fixed mapping relationship between data source structure information and business terms, so that it can be directly read and reused in subsequent natural language queries.

[0037] The natural language interaction module is used to receive natural language statistical analysis requests initiated by users based on the semantic mapping layer;

[0038] The program generation and verification module is used to inject the semantic mapping layer with a preset technical constraint configuration. Through the technical constraint configuration, the large language model is forced to generate an analysis program for an imperative programming language. After generation, the module automatically performs forced verification by parsing the generated code and constructing an abstract syntax tree according to the technical constraint configuration. The technical constraint configuration is preset in a structured configuration format and includes multi-turn dialogue requirement disambiguation rules. These rules are used to determine whether a requirement conflict occurs in a multi-turn dialogue scenario based on the main field set of the requirement output and to identify a unique execution requirement.

[0039] The program execution module is used to run the validated analysis program in an execution environment based on container technology and output the analysis results. The structured output of the analysis results embeds a clear requirement description returned by the large language model. The clear requirement description is a deterministic requirement that is satisfied by this code execution after processing by disambiguation rules.

[0040] The code persistence and reuse module is used to extract the explicit requirement description returned by the large language model from the structured output of the analysis results, establish an association between the requirement description and the corresponding analysis program, and persist the description. When the natural language interaction module receives a new natural language statistical analysis requirement, it calculates the similarity between the new requirement and the persistently saved requirement description using the embedded vector embedding cosine similarity algorithm. If the similarity is greater than a preset threshold, it is determined to be the same requirement, and the associated analysis program is directly called for execution without having to call the large language model again to regenerate the code. The preset threshold is a configurable parameter between 0.85 and 0.95.

[0041] Furthermore, the system is deployed on the following hardware architecture: an application server for deploying the natural language interaction module, the program generation and verification module, the program execution module, and the code persistence and reuse module; a metadata server for persistently storing the semantic mapping layer and its relationships; a data source connection layer for connecting to relational databases and reading table files; and a large language model service node for providing model inference capabilities.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. Eliminate semantic drift: The semantic mapping layer is independently and persistently stored, allowing direct reuse across queries. It establishes a fixed mapping across three layers: data source, table, and field, avoiding semantic inconsistencies caused by repeated parsing of large language models. The semantic mapping layer is detached from the lifecycle of a single query and is specifically designed for code generation scenarios, achieving true semantic asset accumulation.

[0044] 2. Breakthrough in expressive power: Generates complete imperative programs instead of declarative SQL, can call any data analysis library, and supports complex analysis tasks such as moving average, inflection point detection, sentiment analysis, and migration matrix, breaking through the expressive power boundaries of SQL.

[0045] 3. Achieving determinism in analytical logic generation: By setting the temperature parameter to 0, using a fixed random seed, and employing rigid constraints to verify the closed loop, the large language model is transformed from a probabilistic generative model into a deterministic program compiler. Under the same inference environment, the same input can always generate functionally equivalent programs, and the analysis results are reproducible and auditable.

[0046] 4. Supports persistent code reuse, forming accumulative code assets: The large language model embeds explicit, disambiguation-processed requirement descriptions within its structured output. The system associates these descriptions with code and persists them. When new requirements arrive, the system calculates their similarity to the persisted requirement description using an embedded vector embedding cosine similarity algorithm. If the similarity exceeds a preset threshold, the validated code is directly reused. The entire process eliminates the need to call the large language model, forming an efficient closed loop of "requirement definition → code generation → requirement output → associated storage → similarity matching → code reuse".

[0047] 5. With continued use of the system, the semantic assets of the semantic mapping layer and the code assets of the code library are accumulated synchronously, forming a data analysis asset system that is exclusive to the enterprise and can be reused.

[0048] 6. The leap from code generation to enterprise-level application generation: The generated code incorporates database performance optimization strategies, multi-data source collaboration mechanisms, and robust exception handling logic. This effect stems from the unique design in the technical constraint configuration that transforms expert experience into rigid behavioral constraints, rather than relying on the unstable emergent capabilities of the large language model itself.

[0049] 7. Overcoming the limitations of unstructured output from large language models: By embedding structured output format definitions and post-execution format validation, large language models are forced to output structured analysis results, and explicit requirement descriptions are embedded in the structured output, realizing a direct conversion from natural language queries to standardized, visualized analysis reports and reusable requirement tags.

[0050] 8. Effective suppression of context expansion in multi-turn interactions, significantly reducing token costs: In multi-turn interaction scenarios, each turn only needs to transmit the user's historical needs and current needs, maintained in the form of structured variables, without needing to send back historical analysis results. This invention maintains the dialogue state (such as referents and temporary filtering conditions) in the system session as structured variables, rather than sending it back to the large language model in the form of natural language text, thereby achieving a context length that does not grow linearly with the number of interaction turns. The effective suppression of context length allows the system to support continuous interaction in any number of turns, without being limited by the context window of the large language model. At the same time, the multi-turn dialogue requirement disambiguation rule judges based on the main field set and provides a deterministic disambiguation strategy when requirement conflicts occur in multi-turn dialogues, ensuring the uniqueness and accuracy of the execution requirements.

[0051] 9. Eliminate technical barriers: The three-layer mapping allows users to describe the complete data location using pure business terminology, without needing to understand the technical details.

[0052] 10. Semantic assets can be accumulated: The semantic mapping layer supports version management, incremental updates, and user natural language correction, forming a growing and traceable enterprise semantic asset.

[0053] 11. Dual protection of execution security and determinism: Through function signature constraint verification, abstract syntax tree verification, dependency checking, retry mechanism, container isolation execution and output format verification, a multi-layered protection system is formed from code generation to secure execution.

[0054] This invention can be applied to scenarios including but not limited to enterprise data platforms, business intelligence systems, and data service platforms, helping business personnel to complete data statistical analysis independently without any development work, significantly improving data analysis efficiency. The system can be deployed in a private cloud environment or on a local server, ensuring enterprise data security. Furthermore, the versioned management mechanism of the semantic mapping layer makes it an important component of the enterprise data governance system, forming a scalable and traceable enterprise semantic asset. Attached Figure Description

[0055] Figure 1 This is an overall flowchart of the method of the present invention.

[0056] Figure 2 This is a schematic diagram of the three-layer mapping structure of the semantic mapping layer in this invention.

[0057] Figure 3 This is a schematic diagram of the triple mandatory verification closed loop configured with technical constraints in this invention.

[0058] Figure 4 This is a flowchart of code generation and secure execution in this invention.

[0059] Figure 5 This is a module architecture diagram of the system of the present invention.

[0060] Figure 6 This is a graph comparing the token consumption of the present invention and traditional solutions in multiple rounds of interaction.

[0061] Figure 7 This is a flowchart illustrating the code persistence and reuse process in this invention.

[0062] Figure 8 This is a hardware deployment architecture diagram of the system of the present invention. Detailed Implementation

[0063] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0064] like Figure 5 As shown, the self-service data statistical analysis system of the present invention includes a data access module, a semantic mapping management module, a natural language interaction module, a program generation and verification module, a program execution module, and a code persistence and reuse module.

[0065] Example 1: Single data source analysis based on semantic mapping layer

[0066] This embodiment demonstrates the complete process of generating an analysis program based on a semantic mapping layer.

[0067] Step 1: Data Source Connection. Users select the sales data table in the MySQL database through the system interface. The system reads the database table structure and obtains the data source name, table name, field names, and type information.

[0068] Step 2: Semantic Mapping Layer Construction. The system constructs a semantic mapping layer that is independent of the lifecycle of a single query and persists it independently. For example... Figure 2 As shown, a three-tiered mapping relationship is established from technical names to business terms, including the mapping between data source names, table names, and field names and their corresponding business terms. For example, the technical data source name "db_order" is mapped to the business term "order data source," the technical table name "orders" is mapped to "order table," and the field name "sales_amount" is mapped to "sales amount." Users can correct the mapping results through natural language interaction, and the system updates the semantic mapping layer in real time and increments the version number.

[0069] Step 3: Natural Language Analysis Interaction. The user inputs their natural language request: "Statistics on the sales revenue of each product."

[0070] Step 4: Deterministic program generation based on rigid technical constraint configuration. The technical constraint configuration file is loaded as a rule object in memory during system startup. For example... Figure 3 As shown, the technical constraints are configured to form a triple-forced verification loop before, after, and after program generation, including function signature constraint verification, abstract syntax tree parsing verification, and output data structure verification. The system reads the semantic mapping layer from the metadata database and combines it with rule objects through template concatenation: function signature constraints are converted into function definition templates for the target programming language; code generation strategy constraints are converted into conditional generation instructions containing optimization instructions such as "when grouping aggregation or multi-table join operations are needed, embed and execute structured query language in the generated imperative program to process the initial data in the database"; output data structure constraints are converted into an output verification rule set and a structured output format definition containing preset fields for carrying explicit requirement descriptions; and multi-turn dialogue requirement disambiguation rules are converted into requirement disambiguation instructions. The system sets the model temperature parameter to 0 and calls the large language model.

[0071] Under the constraints of the aforementioned technical configuration, the large language model identifies the data source as a relational database. Based on the code generation strategy constraints, it generates an imperative programming language program that prioritizes the use of structured query language. This program executes aggregate queries through database connections to obtain summary sales data for each product. Following the requirements of the output data structure constraints, it encapsulates the results into a standardized format and embeds a clear requirement description, "statistics on sales revenue for each product," within the preset fields of the structured output.

[0072] After generation, the system automatically performs mandatory validation based on technical constraints by parsing the generated code and constructing an abstract syntax tree to verify whether the function signature conforms to preset specifications. After program execution, the system further validates the output to ensure it conforms to the structured output format definition and contains the explicitly stated requirements description. Once all validations pass, the system outputs standardized results including an analysis title, summary text, structured analysis data, visualization information, and the explicitly stated requirements description.

[0073] Step 5: Verification and Execution. For example... Figure 4 As shown, the system performs abstract syntax tree parsing and validation, as well as dependency checks, on the generated program. After successful validation, it executes in a container-based execution environment. The execution environment is configured with resource limits, function call blacklists, and network isolation strategies. The program outputs structured analysis results, which embed an explicit requirement description returned by the large language model.

[0074] The reuse of the semantic mapping layer is demonstrated when a user inputs a new natural language request: "Sales trend statistics by month." The system reads the same semantic mapping layer from the metadata database, repeats the above process, and generates a new analysis program. The semantic mapping layer does not need to be rebuilt, and the user does not need to repeatedly describe the mapping information.

[0075] The multi-turn interaction context management is reflected in the fact that when users perform multiple rounds of analysis consecutively, the system only transmits the semantic mapping layer, the user's historical needs maintained in the form of structured variables, and the current user needs to the large language model in each round, without needing to send back the historical analysis results. The context length only increases slightly in the first and second rounds, and then tends to stabilize, not increasing linearly with the number of interaction rounds.

[0076] The disambiguation rules for multi-turn dialogues are implemented as follows: In a multi-turn dialogue scenario, the system parses the main output field sets corresponding to the user's current and previous requirements. When the user inputs "Statistics on sales revenue for each product" in the first round, the main output field set is {product, sales revenue}; when the user inputs "Statistics on sales revenue for each product category" in the second round, the main output field set is {product category, sales revenue}. The system compares and determines if a requirement conflict has occurred. It uses the last output field set {product category, sales revenue} as the basis to generate the current execution requirement. The large language model embeds the explicit requirement description "Statistics on sales revenue for each product category" in the structured output. When the user inputs "Only view East China region," the current requirement has no independent main output field set, only a limiting condition. The system determines that no requirement conflict has occurred, and that the dimension limited by this limiting condition is semantically related to the current output field set. It then incorporates this limiting condition into the current execution requirement. The large language model embeds the explicit requirement description "Statistics on sales revenue for each product category in East China region" in the structured output. When the system detects an unavoidable semantic conflict between the current requirement and the preceding requirement, such as when the main field sets of the current requirement and the preceding requirement are different, and the dimensions in the limiting conditions of the preceding requirement are not related to the data analysis dimensions corresponding to the current main field set, the requirement cannot be merged, and the system directly returns a failure, with the reason for the failure being that it cannot determine the user's exact meaning.

[0077] The manifestation of code persistence and reuse mechanisms: such as Figure 7 As shown, after the first round of "statistics on sales revenue for each product" is successfully executed, the system extracts the explicit requirement description "statistics on sales revenue for each product" returned by the large language model from its structured output, establishes an association between it and the generated analysis program, and saves it persistently. When the user subsequently inputs "help me check the sales revenue for each product," the system calculates the similarity between the new requirement and the persistently saved requirement description "statistics on sales revenue for each product" using the embedded vector embedding cosine similarity algorithm. If the similarity is greater than a preset threshold (0.9 in this embodiment), it is determined to be the same requirement, and the associated analysis program is directly called for execution, without needing to call the large language model at all. This forms a complete closed loop of "requirement determination → code generation → requirement output → association storage → similarity matching → code reuse."

[0078] Example 2: Complex Statistical Analysis – Inflection Point Detection

[0079] This embodiment demonstrates the fundamental difference between the present invention and the traditional NL2SQL scheme in terms of complex statistical analysis capabilities.

[0080] Business scenario: A retail company wants to analyze monthly sales data over the past three years and automatically identify the "inflection point month" for sales growth.

[0081] Steps 1-2 are the same as in Example 1. Step 3: User input: "Analyze the monthly sales data from January 2023 to December 2025, find the inflection point of the sales growth curve, and output the sales comparison data for the three months before and after the inflection point." Step 4: The system reads the semantic mapping layer and injects the technical constraint configuration. Since inflection point detection requires second-order difference calculation, which exceeds the expressive power of the declarative query language, the large language model generates an imperative program that calls a scientific computing library for numerical calculation under the constraints of the technical constraint configuration. Step 5: The program successfully executes in a secure execution environment, outputting inflection point information and comparison data. Its structured output embeds the explicit requirement description returned by the large language model: "Analyze the inflection point and comparison data of monthly sales from January 2023 to December 2025." This requirement description is associated with the program and persistently saved.

[0082] Example 3: Unstructured Text Analysis – Sentiment Analysis of Product Reviews

[0083] This embodiment demonstrates the present invention's ability to analyze unstructured data.

[0084] Business Scenario: An e-commerce platform operator wants to analyze product review data and count the number of positive, negative, and neutral reviews for each product.

[0085] Step 1: The system accesses the product review table in the MySQL database. Key fields include product name, review text content, and review date. Step 2: The system constructs a semantic mapping layer, where the review text content field is labeled as text. Step 3: The user inputs a natural language requirement, requesting sentiment analysis. Step 4: The system injects technical constraint configuration into the semantic mapping layer. The field type information enables the large language model to automatically identify the need for natural language processing analysis. Under these constraints, the large language model generates a program that calls a natural language processing library to perform sentiment polarity analysis. Step 5: The program successfully executes in a secure execution environment, outputting the number of positive, negative, and neutral reviews for each product, as well as the negative rate. Its structured output embeds the explicit requirement description returned by the large language model: "sentiment analysis of reviews for each product."

[0086] Example 4: Joint Analysis of Multiple Data Sources

[0087] Users access two data sources: a sales order table (MySQL) and a product information table (spreadsheet file). The system builds independent semantic mapping layers for each data source. The user inputs: "Calculate the total sales revenue for each product category." The system simultaneously injects the semantic mapping layers from both data sources into the technical constraint configuration. The code generation strategy constraint in the technical constraint configuration includes a multi-data source step-by-step collaboration strategy. Based on this strategy, the large language model identifies the correlation analysis involving the two data sources in the requirement, generates a program that processes them separately and then merges them for calculation, completing the cross-source analysis.

[0088] Example 5: Comparison Experiment of Token Consumption in Multiple Rounds of Interaction

[0089] This embodiment verifies the effectiveness of the present invention in solving the context inflation problem and its token consumption advantages in multi-turn interaction scenarios through comparative experiments. Figure 6 As shown, the input tokens of the traditional scheme increase from 165 tokens in the first round to 9649 tokens in the 50th round. The token consumption of the scheme of this invention remains stable after the first round, with only 1700 tokens in the 50th round, saving about 75% of the total consumption over 50 rounds.

[0090] Experimental Environment: The large language model uses Deepseek. The model temperature parameter in this invention is set to 0, while traditional solutions use default settings. Traditional NL2SQL solutions use the Vanna framework to generate declarative SQL queries, returning historical dialogue content to the model in each round to maintain context. This invention employs a persistent semantic mapping layer independent of the single query lifecycle and rigid technical constraints pre-configured in a structured format to generate an imperative Python analysis program. In each round, only the semantic mapping layer, historical user requirements maintained in structured variables, and current user requirements are passed; no historical analysis results need to be returned. The test dataset contains four relational tables: customers, orders, products, and employees. The test query set contains 50 statistical analysis query requests with increasing complexity.

[0091] Experimental results show that the traditional scheme increases the number of tokens input per round from 165 tokens in the first round to 9649 tokens in the 50th round, an increase of approximately 58.5 times, with a total input of approximately 238,000 tokens over 50 rounds. The present invention consumes 698 tokens in the first round due to loading the semantic mapping layer, but the input tokens in subsequent rounds only increase slightly in the first and second rounds, then stabilize, reaching only 1700 tokens in the 50th round, which is 17.6% of the traditional scheme's consumption in the same round; the total input over 50 rounds is approximately 59,550 tokens, saving approximately 75%. The present invention's first-round overhead is surpassed by the traditional scheme in the 5th round, and its advantage continues to expand with each additional round of interaction. Furthermore, the output of the present invention in each round is an executable imperative Python program; all programs generated in all 50 rounds pass validation and execute correctly, and each round's structured output embeds a clear requirement description returned by a large language model.

[0092] Example 6: Verification of Code Generation Stability and Persistent Reuse

[0093] This embodiment verifies the performance of the present invention in terms of code generation stability and persistent reuse through repeatability testing.

[0094] Experimental Environment: The large language model used is Deepseek, with the model temperature parameter set to 0 and the random seed set to 42. Test data consisted of a sales data table in a MySQL database, containing 21 fields, with 50,000 randomly generated test data entries. The semantic mapping layer pre-constructed the complete structural information of the data table into a structured semantic description JSON and persisted it. Technical constraint configuration loaded a pre-set configuration file in a structured configuration format at system startup. The user requirement was a complex cross-dimensional analysis task. The retry mechanism allowed a maximum of 3 retries. The similarity threshold for the code persistence and reuse module was preset to 0.9.

[0095] Testing Methodology: Based on the semantic mapping layer, user requirements, and technical constraints, the system calls a large language model to generate an imperative Python analysis program containing the fixed-format function `execute_analysis`. After generation, an abstract syntax tree is validated and a trial run is performed. If it fails, it is retried up to three times. The system runs in a secure execution environment based on container technology. The structured output of the program embeds a clear requirement description returned by the large language model. The system extracts this requirement description, associates it with the program, and persists it. The system evaluates whether the returned results contain the core content of the user requirements.

[0096] Test results: In 20 repeated tests, code was successfully generated 20 times, with a code generation success rate of 100%; it was executed correctly 20 times, with a correct execution success rate of 100%; and it failed 0 times, with a failure rate of 0%.

[0097] Verification of code persistence and reuse: In subsequent tests, a query was initiated using a new requirement that was semantically similar to the already persisted requirement but expressed differently. The system calculated the similarity using the embedded vector embedding cosine similarity algorithm and successfully matched the new requirement with the already persisted explicit requirement description (the similarity was greater than 0.9). The system then directly called the associated analysis program to execute the query. The entire process did not require calling the large language model, and the number of times the large language model was called was zero.

[0098] Example 7: System Hardware Deployment Architecture

[0099] like Figure 8As shown, the system of this invention is deployed in a hardware architecture consisting of an application server, a metadata server, a data source connection layer, a large language model service node, and a container-based execution environment. Specifically, the application server deploys a natural language interaction module, a program generation and verification module, a program execution module, and a code persistence and reuse module, responsible for receiving user requests, configuring scheduling constraints, managing the code generation and verification process, and code reuse; the metadata server is deployed independently to persistently store the semantic mapping layer and verified analysis programs and their associated requirement descriptions, supporting version management and incremental updates; the data source connection layer is configured with read-only database accounts and connects to relational databases through standard database connection protocols; the large language model service node can deploy local private models or call cloud services through encrypted APIs to provide model inference capabilities; the execution environment is implemented based on container technology, configured with resource limits, network isolation, and function call blacklists, forming a secure isolation boundary between the execution environment and the application server and model service node.

[0100] The aforementioned hardware architecture ensures that the system can operate as a standalone enterprise-level data analytics platform, with high availability and data security.

Claims

1. A self-service data statistical analysis method based on artificial intelligence and natural language processing, characterized in that, Includes the following steps: Step 1: Connect to one or more data sources; Step 2: Construct and persist a semantic mapping layer that is independent of the lifecycle of a single natural language query. The semantic mapping layer establishes a fixed mapping relationship between data source structure information and business terms, so that it can be directly read and reused in subsequent natural language queries. The semantic mapping layer is used as an intermediate mapping layer between natural language statistical analysis requirements and executable data analysis programs. Step 3: Receive the natural language statistical analysis request initiated by the user based on the semantic mapping layer; Step 4: Inject the semantic mapping layer with a preset technical constraint configuration. Through the technical constraint configuration, the large language model is forced to generate an analysis program for an imperative programming language. The technical constraint configuration is preset in a structured configuration format. After the program is generated, the system automatically performs forced verification on the generated code according to the technical constraint configuration. Programs that do not conform to the constraints are rejected from entering the execution stage. This eliminates programs that do not conform to the preset technical constraints during the generation stage, ensuring the security and isolation of the subsequent execution environment. Step 5: Run the validated analysis program in the execution environment and output the analysis results.

2. The method according to claim 1, characterized in that, In step 5, the structured output of the analysis results contains an explicit requirement description returned by the large language model. The explicit requirement description is a deterministic requirement that is satisfied by this code execution after processing by disambiguation rules.

3. The method according to claim 2, characterized in that: The semantic mapping layer establishes a three-tiered mapping relationship between data source name, data table name, field name, and business terminology; the semantic mapping layer persists storage in a structured format and supports version management, incremental updates, and version rollback; the data types of the fields include any one or more of numeric, text, date / time, boolean, and categorical types.

4. The method according to claim 1, characterized in that: The semantic mapping layer is stored independently in the metadata database.

5. The method according to claim 1, characterized in that, The technical constraint configuration described in step 4 is pre-configured in the system in the form of a structured configuration file and loaded into a rule object in memory when the system starts. When step 4 is executed, the system converts each constraint in the rule object into a corresponding generation instruction and verification rule.

6. The method according to claim 5, characterized in that, The technical constraint configuration includes at least the following constraints: Function signature constraints are used to limit the analysis program generated by the large language model to have preset standardized function entry and exit points, which are then forcibly verified by the abstract syntax tree parsing and verification after generation. The code generation strategy constraints include preset code generation optimization instructions for different data source types. For relational data sources, the preset optimization instructions include at least the following: when grouping aggregation or multi-table join operations need to be performed, embedding and executing structured query language in the generated imperative program to process the initial data in the database, so as to reduce data transmission overhead and the loop processing cost of imperative language. The output data structure is constrained and has a pre-defined structured output format. After the program is executed, the system performs format verification. Results that do not conform to the structured output format definition are intercepted and a retry is triggered. The structured output format definition includes pre-defined fields for carrying the explicit requirement description. Multi-turn dialogue requirement disambiguation rules are used in multi-turn dialogue scenarios to determine whether requirement conflicts occur based on the main field set of requirement output, and to identify the unique execution requirement.

7. The method according to claim 6, characterized in that, The multi-turn dialogue disambiguation rules include: the system parses the output main field sets corresponding to the current user's request and the previous requests, and determines whether a request conflict occurs by comparing the output main field sets; when multiple different output main field sets appear, the last output main field set is used, and the constraints in the previous requests that are compatible with the last output main field set are merged into the current execution request; the compatible constraints refer to the dimension defined by the constraint being semantically related to the data analysis dimension corresponding to the last output main field set; when an unavoidable semantic conflict is detected, the system directly returns failure, with the reason for failure being that the user's accurate meaning cannot be determined.

8. The method according to claim 6, characterized in that, In the code generation strategy constraints, for relational data sources, the preset optimization instructions also include: requiring the generated structured query language to follow the rule of using aggregate functions to avoid loops, and the rule of using backticks to escape table names and field names.

9. The method according to claim 6, characterized in that, The structured output format definition embedded in the output data structure constraint includes a variety of preset analysis result type definitions, each of which includes a visualization field for indicating the front-end visualization format; the technical constraint configuration also includes automatic inference rules from data features to the various analysis result types.

10. The method according to claim 5, characterized in that, In step 4, the system combines the semantic mapping layer and the rule objects in memory through template concatenation, and converts them into the input of the large language model. Under the dual constraints of the fixed semantic constraints provided by the semantic mapping layer and the program structure constraints provided by the rule objects, the large language model generates an imperative programming language analysis program.

11. The method according to claim 1, characterized in that: When calling the large language model, the model temperature parameter is set to 0 and the random seed is fixed to ensure that the same input produces the same analysis program under the same inference environment.

12. The method according to claim 1, characterized in that, The steps following step 4 and before step 5 include code verification and retry steps: The generated analysis program is subjected to syntax validation and dependency checks; The syntax verification uses an abstract syntax tree parsing method to check whether the code conforms to the basic syntax rules of the target programming language; The dependency check confirms that the modules, classes, or functions referenced by the analyzer are available in the execution environment; If the verification fails, the error message will be sent to the large language model and a regeneration will be triggered, with a maximum of a preset number of retries.

13. The method according to claim 1, characterized in that: The execution environment is configured with resource limits, function call blacklists, and network isolation policies; the execution environment is implemented based on container technology.

14. The method according to claim 1, characterized in that, The construction and modification of the semantic mapping layer in step 2 includes: Construction phase: The semantic mapping layer is initially constructed using any one or more of the following methods: automatic parsing based on the data source structure by the system, manual annotation by the user through the interface, definition by the user through natural language description, or automatic inference by a large language model based on the data source structure; Correction and evolution process: Users can supplement or correct the mapping results through natural language interaction. The corrected semantic mapping layer will automatically update the version number and retain the historical version.

15. The method according to claim 1, characterized in that: When multiple data sources are connected, each data source establishes an independent connection, and the system manages the access permissions of each data source independently to ensure data isolation between data sources. When multiple data sources need to be correlated, the system automatically identifies the fields that can be correlated through the field mapping in the semantic mapping layer.

16. The method according to claim 2, characterized in that, Step 5 is followed by a code persistence and reuse step: the system extracts the explicit requirement description returned by the large language model from the structured output of the analysis results, establishes an association between the requirement description and the corresponding analysis program, and persists it; when the system receives a new natural language statistical analysis requirement, it calculates the similarity between the new requirement and the persisted requirement description using the embedded vector embedding cosine similarity algorithm. If the similarity is greater than a preset threshold, it is determined to be the same requirement, and the associated analysis program is directly called for execution without having to call the large language model again to regenerate the code; the preset threshold is a configurable parameter between 0.85 and 0.

95.

17. A self-service data statistical analysis system based on artificial intelligence and natural language processing, characterized in that, include: The data access module is used to access one or more data sources; The semantic mapping management module is used to build and persist the semantic mapping layer that is independent of the lifecycle of a single natural language query, and establish a fixed mapping relationship between data source structure information and business terms, so that it can be directly read and reused in subsequent natural language queries. The natural language interaction module is used to receive natural language statistical analysis requests initiated by users based on the semantic mapping layer; The program generation and verification module is used to inject the semantic mapping layer with a preset technical constraint configuration. Through the technical constraint configuration, the large language model is forced to generate an analysis program for an imperative programming language. After generation, the generated code is automatically verified according to the technical constraint configuration. The technical constraint configuration is preset in a structured configuration format and includes multi-turn dialogue requirement disambiguation rules. The multi-turn dialogue requirement disambiguation rules are used to determine whether a requirement conflict occurs based on the main field set of the requirement output in a multi-turn dialogue scenario, and to determine the unique execution requirement. The program execution module is used to run the validated analysis program in an execution environment based on container technology and output the analysis results. The structured output of the analysis results embeds a clear requirement description returned by the large language model. The clear requirement description is a deterministic requirement that is satisfied by this code execution after processing by disambiguation rules. The code persistence and reuse module is used to extract the explicit requirement description returned by the large language model from the structured output of the analysis results, establish an association between the requirement description and the corresponding analysis program, and persist the description. When the natural language interaction module receives a new natural language statistical analysis requirement, it calculates the similarity between the new requirement and the persistently saved requirement description using a similarity algorithm. If the similarity is greater than a preset threshold, it is determined to be the same requirement, and the associated analysis program is directly called for execution without having to call the large language model again to regenerate the code.

18. The system according to claim 17, characterized in that: The system is deployed on the following hardware architecture: an application server, used to deploy the natural language interaction module, the program generation and verification module, the program execution module, and the code persistence and reuse module; A metadata server is used to persistently store the semantic mapping layer and its relationships; a data source connection layer is used to connect to relational databases and read table files. Large language model service nodes are used to provide model inference capabilities.

Images