Integrated Text to SQL (Structured Query Language) and report generation method based on knowledge graph

Abstract

The invention discloses an integrated Text to SQL (Structured Query Language) and report generation method based on a knowledge graph, and aims to solve the problems that the existing data technology lacks cross-industry universality, is low in query efficiency and the like. According to the method, firstly, a cross-industry general entity association network is constructed based on a knowledge graph, entity ambiguity is accurately analyzed, and general semantic support is provided for mapping a natural language to a service database; secondly, designing a multi-scene adaptive Text to SQL core reasoning module, deeply fusing general service logic and semantic understanding, and breaking a cross-domain landing bottleneck; finally, a self-adaptive data analysis module is developed in a matched mode, data multi-source heterogeneity and real-time updating challenges are effectively overcome by dynamically adjusting data association weights and optimizing a cross-source query path, and low-cost and high-efficiency calling of historical business data is achieved. According to the scheme, multi-dimensional requirements of daily operation, decision support and the like are comprehensively met, and an efficient and flexible general data intelligent solution is provided for each industry.

Description

Technical Field

[0001] This application relates to the fields of computer data processing and natural language processing, and in particular to an integrated Text to SQL and report generation method based on knowledge graphs. Background Technology

[0002] Text-to-SQL technology, as a core support for natural language and database interaction, aims to convert user-input natural language queries into executable SQL statements, enabling efficient access and data extraction from structured databases. This technology is widely used in data querying, operational analysis, and decision support across various industries, and is a key technology for lowering the barrier to data use and improving data utilization efficiency.

[0003] For example, in daily business operations, marketing personnel need to query product sales data for different regions in a given quarter, and finance personnel need to compile monthly income and expenditure details. Traditionally, this requires professional data analysts to write SQL statements to complete the queries, which not only incurs high communication costs and slow response times but may also lead to inaccurate results due to misunderstandings of the requirements. Text-to-SQL technology, however, allows non-technical personnel to directly express their query needs using natural language, automatically generating corresponding SQL statements and returning the results. This greatly simplifies the data query process, improves work efficiency, and helps enterprises achieve rapid data-driven decision-making.

[0004] In the process of developing this application, the inventors discovered the following shortcomings in the relevant technologies: existing Text to SQL algorithms are mostly developed based on general natural language processing models, lacking the ability to accurately identify cross-industry business entities, and are easily affected by ambiguous entities and technical terms, resulting in low SQL generation accuracy; at the same time, the algorithms do not fully integrate business knowledge and database structure information, making it difficult to handle complex relational query scenarios, and have poor adaptability to multi-source heterogeneous databases. The overall generated SQL statements have problems such as syntax errors and incomplete logic, which cannot meet the high-precision data query needs of practical applications in various industries. Summary of the Invention

[0005] This application provides an integrated Text to SQL and report generation method based on knowledge graphs. The technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a multi-dimensional preprocessing method for natural language queries, the method comprising:

[0007] It receives general domain natural language queries from users, performs basic preprocessing operations on the query text such as word segmentation, stop word removal, and grammatical error correction, removes invalid information, and improves the standardization of the text;

[0008] The system utilizes a general-domain entity extraction model, leveraging its multi-scenario adaptability optimized based on cross-industry corpus pre-training, to accurately identify and extract core business entities from queries, including key elements such as object names, data metrics, business terms, and process nodes. Ambiguous business entities are masked by replacing their content with specific mask identifiers (such as [ENTITY_MASK]), preventing ambiguity from interfering with subsequent SQL generation logic and reducing accuracy loss.

[0009] The output includes preprocessed unambiguous query text and a set of core business entities, providing high-quality input data support for subsequent SQL generation.

[0010] This process fully leverages the cross-industry adaptability and ambiguity handling mechanism of the dedicated entity extraction model. Through standardized preprocessing, it effectively improves the structuring level of natural language queries, laying a reliable foundation for subsequent accurate mapping to the database.

[0011] Secondly, embodiments of this application also provide a SQL generation optimization module based on RAG and knowledge graph:

[0012] A general domain-specific entity extraction model is invoked to perform secondary precise entity extraction on the preprocessed query text, further correcting the entity recognition results, and rewriting the original query with entity mask based on the recognition results to form standardized query text;

[0013] The RAG (Retrieval-Augmented Generation) retrieval mechanism is activated. Based on the extracted core business entities, relevant structured data indexes (including table structure, field meaning, data relationship, etc.) in the general domain database are retrieved to establish a preliminary connection between the query and the database.

[0014] Simultaneously, the constructed general business entity knowledge graph is retrieved, and the accurate database association content (including standard field names, association table information, etc.) corresponding to the masked entity is obtained through entity similarity matching algorithms (such as cosine similarity calculation), and the masked entity is accurately replaced.

[0015] The rewritten query text, the data index obtained from RAG retrieval, and the entity association content matched by the knowledge graph are all input into the large model. Through the semantic understanding and logical reasoning capabilities of the large model, an executable SQL statement for the target database is generated.

[0016] An SQL syntax validation mechanism is introduced to perform syntax rule validation and logical conflict detection on the generated SQL statements, automatically correcting syntax errors, field matching errors, and other issues to ensure the executability of the SQL statements.

[0017] This module effectively solves problems such as entity ambiguity and database structure mismatch by leveraging both RAG retrieval and knowledge graphs, improving the accuracy and rationality of SQL generation and providing reliable support for complex query scenarios.

[0018] Thirdly, this application also provides a multi-source heterogeneous database adaptive adaptation and SQL intelligent error correction module:

[0019] We build a general-purpose database metadata parsing engine that supports automatic parsing of metadata for various types of databases, including relational databases and distributed databases. It extracts core metadata such as table structure, field types, primary key and foreign key relationships, and index information, providing underlying data structure support for SQL error correction.

[0020] Based on the parsed metadata and cross-database syntax rule base, an intelligent SQL syntax verification mechanism is established to perform multi-dimensional error correction on the statements generated by Text to SQL, including field name spelling errors correction, table relationship legality verification, function parameter matching optimization, data type compatibility adjustment, and syntax format standardization, so as to comprehensively avoid execution risks at the syntax level.

[0021] We designed a context-aware SQL semantic error correction model that combines business scenarios with metadata association logic to intelligently correct issues such as semantic ambiguity, logical contradictions, and deviations from query objectives. For example, it automatically completes missing query conditions, adjusts unreasonable aggregate function usage, and optimizes the table join order of related queries, ensuring that SQL statements meet business logic requirements while maintaining syntactic correctness. We also established a dynamic error correction feedback mechanism that links the error correction process with the database access interface, completing final verification and optimization before SQL statement execution, and recording error correction logs to provide data support for subsequent iterative optimization of the Text to SQL algorithm.

[0022] This module significantly improves the accuracy and execution reliability of generated SQL statements by implementing key steps such as metadata-driven syntax validation, semantic error correction, and logic optimization, while achieving flexible adaptation to multi-source heterogeneous databases. It breaks through the technical bottlenecks of existing algorithms, which suffer from high syntax error rates and poor semantic adaptability during SQL generation.

[0023] Fourthly, embodiments of this application also provide a training and inference framework for a Text to SQL algorithm, which specifically includes the following:

[0024] We constructed a cross-industry general Text to SQL dataset, covering query samples from multiple scenarios such as operations management, customer service, financial statistics, and market analysis. Each sample contains annotations such as natural language queries, corresponding standard SQL statements, database metadata, and knowledge graph association information.

[0025] The dataset was divided into training, validation and test sets in an 8:1:1 ratio. The transfer learning method was adopted, the model parameters were initialized based on the pre-trained large language model, and the model was jointly trained with the general business knowledge graph.

[0026] During training, a multi-task loss function is introduced, including semantic matching loss (to optimize the semantic consistency between natural language and SQL statements), SQL generation accuracy loss (to optimize the syntax and logical correctness of SQL), and query efficiency loss (to optimize the SQL execution speed). The total loss is calculated by weighted summation, and the model parameters are optimized by backpropagation.

[0027] During the inference phase, the system receives natural language queries input by the user and sequentially processes them through multi-dimensional preprocessing, SQL generation by fusing RAG and knowledge graph, and adaptation to multi-source heterogeneous databases, generating executable SQL statements and submitting them to the target database.

[0028] It receives query results returned by the database, organizes the results in a structured manner according to the query requirements, generates standardized data reports, and supports exporting reports in multiple formats (such as Excel, PDF, JSON) and online previewing.

[0029] Establish a reasoning result feedback mechanism to collect user satisfaction evaluations of SQL generation results and query reports. Add effective feedback data to the training set for model iteration and optimization to continuously improve algorithm performance.

[0030] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0031] This application proposes an integrated Text to SQL and report generation method based on knowledge graphs. The algorithm comprehensively addresses the problems of inaccurate entity recognition, weak ambiguity handling, poor database adaptability, and low SQL generation quality in existing Text to SQL technologies through three core modules: multi-dimensional preprocessing, knowledge graph and RAG fusion optimization, and multi-source heterogeneous database adaptation, combined with a complete training and inference framework.

[0032] First, through multi-dimensional preprocessing of natural language queries, effective masking of ambiguous entities and accurate extraction of core entities are achieved, providing high-quality input for subsequent SQL generation. Second, relying on the fusion and optimization of RAG retrieval and general knowledge graph, business knowledge and database structure information are fully utilized to improve the syntactic correctness and logical integrity of SQL statements. Third, the multi-source heterogeneous database adaptive adaptation module achieves flexible compatibility with different types of databases, broadening the application scenarios of the algorithm. Finally, through multi-task joint training and closed-loop feedback mechanisms, model performance is continuously optimized to ensure high accuracy and efficiency of the algorithm in various industry scenarios.

[0033] This algorithm does not require users to have professional SQL knowledge. It can complete accurate queries on multi-source heterogeneous databases using only natural language, effectively reducing the threshold for data use, improving the efficiency of data query and analysis, and providing efficient and reliable technical support for operation management and decision support in various industries. Attached Figure Description

[0034] Figure 1 This is an overall structural diagram of an integrated Text to SQL and report generation method based on knowledge graphs provided in this application embodiment;

[0035] Figure 2 This is a structural block diagram of a natural language query multi-dimensional preprocessing module provided in an embodiment of this application for natural language question preprocessing;

[0036] Figure 3 This is a structural block diagram of the SQL generation module based on RAG and knowledge graph provided in the embodiments of this application;

[0037] Figure 4 This is a structural block diagram of a data analysis and report generation module provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Exemplary embodiments will be described in detail here, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods consistent with some aspects of this application as detailed in the appended claims.

[0039] In the process of developing this application, the inventors discovered that existing Text to SQL algorithms have three core problems: First, the entity recognition accuracy is insufficient, and general NLP models are difficult to adapt to cross-industry professional terms and ambiguous entities, resulting in field matching errors when generating SQL; second, knowledge integration is insufficient, and the algorithm does not effectively combine business knowledge with database structure information, making it unable to handle complex relational queries; third, the result presentation format is monotonous, only returning raw query data and lacking standardized and visualized analysis reports, requiring users to manually organize and analyze the data, which is inefficient.

[0040] Existing technologies largely rely on the semantic understanding capabilities of a single pre-trained model, treating Text to SQL as a simple "natural language to grammar" task, neglecting the deep adaptation of business knowledge and the user experience of result presentation. Although some solutions introduce retrieval mechanisms, they fail to form a complete closed loop of "preprocessing-generation-reporting-feedback," making it difficult to meet the actual needs of various industries for high-precision queries and efficient decision-making.

[0041] The inventors of this application attempted to systematically solve the above problems through an integrated architecture of "multi-dimensional preprocessing + dual fusion of knowledge graph and RAG + standardized report generation", and found that the architecture can significantly improve the accuracy of SQL generation and the practicality of results, which is a creative effort.

[0042] To address related technical issues, this application provides an integrated TexttoSQL and report generation method based on knowledge graphs. The method includes: first, multi-dimensional preprocessing of the user-input natural language query to accurately extract core entities and resolve ambiguities; then, retrieving database metadata and knowledge graph association information through RAG retrieval to optimize the SQL generation logic; next, using a structured data-driven report generation module to transform the query results into a standardized, visualized analysis report; and finally, ensuring algorithm performance through a complete training and inference framework, supporting result feedback and model iteration.

[0043] As can be seen, this solution, through its end-to-end design of "preprocessing-generation-reporting-feedback", not only leverages the semantic understanding advantages of large models, but also fills the gaps in business knowledge through knowledge graphs and RAGs, while improving the practicality of results through standardized report generation. It effectively solves the core problems of low accuracy and poor result utilization of existing Text to SQL algorithms.

[0044] The following is a description of some terms used in the embodiments of this application.

[0045] RAG (Retrieval-Augmented Generation): A retrieval-enhanced generation technique that supplements model input information by retrieving information from external knowledge bases, thereby improving the accuracy and reasonableness of the generated results.

[0046] Knowledge graph: A knowledge base that stores entities and their relationships in a graph structure. Its core elements include entities, attributes, and relationships, and it is used to resolve entity ambiguities and supplement implicit relationships.

[0047] Text to SQL: A technology that converts user-input natural language queries into executable SQL statements, enabling interaction between natural language and structured databases.

[0048] Standardized Analysis Report: A structured document generated using a unified template based on query results and business scenarios. It includes core indicators, trend charts, anomaly alerts, and decision-making suggestions, and supports export in multiple formats.

[0049] General Text to SQL Dataset: Adopts publicly available cross-industry multi-scenario Text to SQL datasets (such as Spider, WikiSQL extended version), covering 15+ industries and 80,000+ query samples, including annotations of natural language queries, standard SQL, database metadata, knowledge graph association information, etc.

[0050] The overall process provided in the embodiments of this application will be described in detail below with reference to Figure 1 (overall algorithm framework diagram).

[0051] Referring to Figure 1, the integrated Text to SQL and report generation method based on knowledge graphs provided in this application embodiment may include the following steps:

[0052] First, the natural language query undergoes multi-dimensional preprocessing. Users do not need professional SQL knowledge; they only need to express their query requirements in natural language. The system first performs standardized preprocessing on this natural language query, with the core objectives of improving text standardization, accurately extracting entities, and eliminating ambiguity. Understandably, natural language contains issues such as non-standard expressions, entity ambiguity, and redundant information. Directly inputting these into a large model would lead to a decrease in SQL generation accuracy. Therefore, preprocessing is fundamental to ensuring the effectiveness of subsequent processes and requires multi-step standardization to improve input quality.

[0053] The embodiment of this application adopts a three-stage preprocessing process of "basic cleaning - entity extraction - ambiguity processing". First, the jieba word segmentation tool is used to segment the query text, and meaningless stop words such as "de", "le", "qing wen" are removed. Then, a language model is used for grammar correction to correct spelling mistakes and improper grammar expressions such as "xiaoliang" and "paimingci", and a standardized text is output. Subsequently, a general domain-specific entity extraction model is called to accurately identify the core business entities in the query, including time entities (the second quarter of 2024), object entities (products, regions), indicator entities (sales volume), operation entities (statistics, ranking), etc., and an entity set and entity type annotation are output. Finally, entity ambiguity detection is performed based on the general business knowledge graph. If an entity has multiple meanings (e.g., "apple" may refer to a fruit product or an electronic brand), the entity is replaced with the [ENTITY_MASK] mask to avoid interference from ambiguous information in the SQL generation logic, and finally an unambiguous query text and a standardized entity set are output. This preprocessing process combines tool-based cleaning and model-based entity extraction to ensure both text normativity and accurate capture of the core elements of the query, providing high-quality input for subsequent SQL generation.

[0054] Secondly, optimize SQL generation based on RAG and knowledge graph.

[0055] The unambiguous query text and entity set after preprocessing need to be combined with database information and business knowledge for SQL generation optimization. The core goal is to solve problems such as "entity-database field mismatch", "incomplete query logic", and "syntax error". Existing technologies only rely on the general semantic understanding ability of large models and lack adaptation to database structures and business knowledge, resulting in problems such as "field does not exist" and "association logic error" often occurring in the generated SQL statements. This application realizes the optimization of SQL generation by fusing RAG retrieval and knowledge graph to provide accurate external support information for large models.

[0056] Specifically, firstly, a dedicated entity extraction model is invoked to perform a secondary verification of the preprocessed query text, correcting any omissions or misjudgments that may have occurred during the initial extraction. Then, based on the final entity set, the original query is rewritten into a standardized sentence structure of "[time entity][operation entity][object entity] of [indicator entity]" (e.g., "[second quarter of 2024] [statistics] [region] of [product sales] [ranking]"), reducing the semantic understanding difficulty of the large model. Subsequently, the RAG retrieval mechanism is activated, using the core entities as search keywords to retrieve the metadata index of the target database (stored in the vector database FAISS), obtaining the table structure (e.g., product table, sales record table), field information (e.g., product_name, region, sales_volume, sales_date), and primary key and foreign key relationships (e.g., product_id in the sales record table). This module establishes a structural mapping between queries and the database, using elements such as the ID of the associated product table. Simultaneously, it retrieves general business knowledge graphs, calculates the similarity between masked entities and entities in the graph using a cosine similarity algorithm, filters out the standard entities with the highest matching degree, and obtains the database field mapping relationships corresponding to the entities through the knowledge graph. It then precisely replaces masked entities with non-standard entities, generating query text adapted to the database. The rewritten query text, metadata retrieved by RAG, and entity association content matched by the knowledge graph are input into a large model (this application uses a fine-tuned SQLCoder 7B model adapted to general Text to SQL tasks). The large model's semantic understanding and logical reasoning capabilities generate preliminary SQL statements. Finally, the SQLFluff syntax validation tool is introduced to perform syntax rule validation and logical conflict detection on the generated SQL statements, automatically correcting syntax errors, field matching errors, and other issues to ensure the executability of the SQL statements. This module, through the dual empowerment of RAG retrieval and knowledge graphs, effectively solves problems such as entity ambiguity and database structure mismatch, improving the accuracy and rationality of SQL generation.

[0057] Then, standardized report generation is achieved using structured data-driven approaches.

[0058] The structured data obtained after executing SQL statements needs to be transformed into standardized analysis reports. The core objective is to improve data readability and usability, providing direct support for user decision-making. Existing Text to SQL algorithms mostly only return raw query results, requiring users to manually organize, analyze, and visualize them. This is not only inefficient but may also lead to the underutilization of data value due to a lack of professional analytical skills. This application achieves automated transformation from raw data to decision support by constructing a standardized report generation module.

[0059] First, the system receives structured query results from the database and automatically cleans the data, removing outliers, filling in missing values, and standardizing data formats to ensure data quality. Then, based on query requirements and business scenarios, it automatically parses data dimensions and performs multi-dimensional analysis, including calculating core statistical indicators such as sums, means, and percentages. For time-series data, it generates trend charts to identify growth / decline trends and uses the standard deviation method to identify outliers and label their causes. Next, based on a configurable general report template, the content is organized logically into "core indicators - detailed data - trend charts - anomaly alerts - decision suggestions." The core indicators section highlights key statistical results, the detailed data section presents categorized data in tabular form, the visualization charts section automatically generates bar charts, line charts, pie charts, etc., the anomaly alerts section labels outlier data and possible causes, and the decision suggestions section generates targeted suggestions based on the data results. Finally, the report supports exporting to common formats such as Excel, PDF, and JSON, and provides an online preview function, preserving the original data traceability link. Users can click on report indicators to view the corresponding original data and SQL query statements, ensuring data verifiability and traceability. This module transforms raw data into high-value decision support materials through automated analysis and structured presentation, significantly improving the practicality and user experience of the Text to SQL algorithm.

[0060] Finally, an efficient training and testing framework for general domain models of Text to SQL is proposed.

[0061] To ensure the performance stability and continuous optimization capability of the algorithm, this application constructs a complete training and inference framework, covering four core stages: data adaptation, model training, inference execution, and result feedback. The core objective of model training is to optimize model parameters through multi-task joint training and improve the accuracy of SQL generation; the core objective of inference execution is to achieve end-to-end automated processing from natural language queries to standardized reports; and the core objective of result feedback is to collect user feedback to achieve continuous iterative optimization of the model.

[0062] We adopted a publicly available, cross-industry, general-purpose Text to SQL dataset (such as the Spider extended version), covering 15+ scenarios including operations management, financial statistics, market analysis, and customer service, with a total of 80,000+ query samples. Each sample contains annotations including natural language queries, corresponding standard SQL statements, database metadata, and knowledge graph association information. The dataset was divided into a training set (64,000 records), a validation set (8,000 records), and a test set (8,000 records) in an 8:1:1 ratio. We employed transfer learning, initializing model parameters based on a pre-trained large language model (SQLCoder 7B), and jointly training the dataset with the training set and a general business knowledge graph. During training, a multi-task loss function was introduced, including calculating the semantic similarity loss between natural language queries and SQL statements (optimizing semantic understanding ability) and the difference loss between generated SQL and standard SQL based on edit distance (optimizing grammatical and logical correctness). The total loss was a weighted sum of the two (with weights of 0.4 and 0.6, respectively). Model parameters were optimized through backpropagation, and model performance metrics (SQL) were monitored in real time using the validation set. The system generates accuracy, grammatical correctness, and query response time metrics. It employs an early stopping strategy (stopping training if performance doesn't improve after 5 consecutive epochs) to prevent overfitting and dynamically adjusts the learning rate (initial learning rate 1e-5, decaying by 10% every 2 epochs). During the inference phase, it receives natural language queries from the user, performs multi-dimensional preprocessing, generates SQL statements that merge RAG and knowledge graphs, produces executable SQL statements for the target database, executes the SQL statements, receives structured data returned from the database, cleans the data, performs multi-dimensional analysis, generates standardized analysis reports, provides online preview and multi-format export capabilities, and preserves the data traceability chain.

[0063] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0064] This application provides an integrated Text to SQL and report generation method based on knowledge graphs. Through standardized design of the entire process and collaborative linkage of multiple modules, it systematically solves the problems of inaccurate entity recognition, low SQL generation quality, poor result utilization, and lack of continuous optimization capability in existing technologies, and has significant technical advantages.

[0065] The preprocessing stage, through a three-stage process of "basic cleaning - entity extraction - ambiguity resolution," effectively improves the structure of natural language queries, eliminates ambiguity interference, and provides high-quality input for SQL generation. The SQL generation stage, leveraging the dual capabilities of RAG retrieval and knowledge graphs, fully integrates database metadata and business knowledge, significantly improving the syntactic correctness, logical integrity, and executability of SQL statements, thus addressing the "knowledge gap" problem in large models. The report generation stage, through automated data cleaning, multi-dimensional analysis, and structured presentation, transforms raw query results into standardized, visual analysis reports, eliminating the need for manual user processing and improving data utilization and decision-making efficiency. The training and inference stage, through public dataset adaptation, multi-task joint training, and closed-loop feedback iteration, ensures the performance stability of the algorithm in different scenarios while continuously optimizing report quality, providing users with end-to-end automated services for "query-analysis-decision-making."

[0066] In summary, this algorithm does not require users to have professional SQL knowledge and data analysis skills. Users can obtain accurate and practical analysis reports simply by using natural language. It effectively lowers the threshold for data use, improves the efficiency of data-driven decision-making, and provides a comprehensive and efficient solution for various industries' operation management, market analysis, financial statistics and other scenarios.

[0067] It should be understood that although the steps have been described above, they are not necessarily to be performed in the order described above. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders. Any reasonable order is within the scope of protection of this application, and will not be described in detail here.

[0068] Moreover, at least some steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] Corresponding to the above method embodiments, this application also provides a structural block diagram of a multi-dimensional preprocessing module for natural language queries. Specifically, as shown... Figure 2 As shown:

[0070] First, since the unstructured text information contained in the user's original natural language query cannot be directly processed by a large model, a Natural Language Processing (NLP) toolchain is needed for preprocessing. Preprocessing mainly includes text standardization and cleaning, core business entity extraction, and ambiguity detection. Each subtask is modeled by a basic cleaning unit, an entity extraction unit, and an ambiguity resolution unit, respectively. The processed unambiguous query text and standardized entity set will be used in subsequent processes to generate SQL statements, thereby improving the accuracy of the Text to SQL algorithm.

[0071] Subsequently, the preprocessed results are input into different functional modules for in-depth processing, and finally the overall transformation is completed through a pipelined processing mechanism. Specifically, the basic cleaning unit integrates word segmentation tools, a stop word dictionary, and a grammar correction model to achieve standardized text cleaning; the entity extraction unit, based on a fine-tuned BERT model, achieves accurate extraction and type labeling of core business entities; and the ambiguity handling unit connects to a general business knowledge graph, detects ambiguities through entity similarity matching, and performs masking processing.

[0072] This process fully leverages natural language processing and knowledge graph technologies. Each unit performs text cleaning, entity extraction, and disambiguation, and a pipelined processing mechanism constructs a complete processing flow. This multi-stage processing not only progressively optimizes the quality of the input text but also enhances the understanding of complex natural language queries, providing more accurate and structured information input for subsequent SQL generation.

[0073] In addition, there is an SQL generation optimization module based on RAG and knowledge graph, as shown in Figure 3.

[0074] The input information refers to the preprocessed query text and entity set, while the knowledge index refers to the target database metadata index stored in the vector database and the entity association information in the knowledge graph. The purpose of this fusion module is to achieve deep fusion of multi-source information, leveraging the retrieval advantages of RAG technology while resolving issues of entity ambiguity and database structure mismatch.

[0075] The entire module's structure revolves around multi-source information collaboration, achieving information processing and fusion through five sub-units. The query rewriting unit performs secondary entity verification and standardized sentence rewriting on the preprocessed text, laying the foundation for subsequent processing; the RAG retrieval unit connects to a vector database, retrieves metadata indexes from the target database, and acquires structured knowledge; the knowledge graph matching unit retrieves entity association information, achieving ambiguous entity replacement and field mapping to enhance semantic understanding; the large model generation unit generates preliminary SQL statements based on multi-source inputs, leveraging the model's generative capabilities; and the syntax verification unit corrects errors using SQL syntax verification tools and outputs standard SQL. The RAG retrieval process uses a cosine similarity algorithm to calculate the semantic matching degree between entities and metadata; the core formula is as follows:

[0076]

[0077] In summary, this fusion module aims to fully leverage the advantages of RAG technology and large models to ensure accurate matching between input information and database structure, thus guaranteeing efficient and accurate Text to SQL conversion.

[0078] Fourthly, embodiments of this application also provide a structural block diagram of the data analysis and report generation module, see details below. Figure 4 :

[0079] The data analysis and report generation module is designed based on the core need for data-driven decision-making. It aims to transform raw structured query results into standardized analytical reports with decision-making value through an automated process. The core input of this module is the structured query results, and the output is a standardized analytical report. It contains four closely cooperating sub-units, and the design and functional implementation logic of each sub-unit are as follows:

[0080] The data cleaning unit draws on anomaly detection algorithms and data completion strategies from statistics. It identifies and removes outliers through box plot analysis, completes missing values ​​using methods such as mean / median imputation, and standardizes the format based on regular expressions and type mapping rules.

[0081] The data analysis module performs analysis operations using a "feature engineering + statistical modeling + machine learning classification" approach. It first constructs derived features such as month-on-month growth rate and percentage contribution, then uses descriptive statistics to summarize indicators, mines trend patterns through time series decomposition, and combines Z-score / Grubbs statistical tests with the LightGBM model to accurately identify outliers. Taking sales data as an example, it can automatically generate quarterly sales comparisons and monthly growth rate trend charts, and mark abnormal fluctuation points based on statistical significance. This improves data utilization and decision-making efficiency, eliminating the need for manual user processing. During the training and inference phases, it ensures the algorithm's performance stability in different scenarios through public dataset adaptation, multi-task joint training, and closed-loop feedback iteration, while continuously optimizing report quality to provide users with end-to-end automated services for "query-analysis-decision-making."

[0082] The report's structured units utilize template engine technology, organizing report content according to a common template. Natural Language Generation (NLG) technology transforms data analysis results into text descriptions, and visualization libraries such as Echarts and Highcharts are used to generate bar charts, line charts, heatmaps, and other visual charts. The templates support custom configuration, allowing users to adjust the report structure and chart types to achieve personalized report output.

[0083] The export and traceability unit utilizes file format conversion technology, supporting export to multiple formats such as PDF, Excel, and Word. Simultaneously, by establishing a data lineage graph, it fully preserves the original data and the SQL traceability chain. During data traceability, a visual interface displays the complete processing path of data from the original table to the query results, including each step of data transformation, calculation logic, and referenced SQL statements, ensuring transparency and traceability of the data processing process.

[0084] This module, through the collaborative work of the four sub-units mentioned above, achieves the automated transformation from raw data to decision support. It not only significantly improves the practicality of the analysis results, but also reduces the errors and costs of manual analysis through standardized processes, providing efficient and reliable data support for business decisions.

[0085] It should be understood that "multiple" as mentioned herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, the step numbers described herein only exemplify one possible execution order between steps. In some other embodiments, the steps may not be executed in the order shown in the numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this. The above embodiments can also be combined arbitrarily, and the combination schemes will not be elaborated here.

[0086] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A knowledge graph-based integrated Text to SQL and report generation method, characterized in that, The core technology framework of the algorithm is built around the entire process of "natural language interaction - data processing - report generation," including two core modules: a Text to SQL algorithm based on a large model and a report generation algorithm driven by structured data. Users do not need to master professional data query language; they can trigger automated data processing workflows simply by expressing general domain-related data questions in natural language. The Text to SQL algorithm module is responsible for converting natural language queries into executable SQL statements, enabling accurate data retrieval from general-purpose databases; The structured data-driven report generation algorithm module is responsible for analyzing and processing the retrieved general-domain structured data and automatically generating standardized data reports. The entire framework achieves end-to-end automated processing from natural language questioning to data report output through the coordinated operation of two core modules, adapting to the general application needs of various industries such as data query, operational analysis, and decision support.

2. The method for general-domain natural language interaction data processing and report generation based on a large model as described in claim 1, characterized in that, The multi-dimensional preprocessing process for natural language queries specifically includes: It receives general-domain natural language queries from users, performs basic preprocessing operations such as word segmentation, stop word removal, and grammatical error correction, removes redundant and interfering information and corrects grammatical expression deviations, significantly improving the standardization and semantic clarity of the query text, and laying a high-quality data foundation for subsequent entity recognition and SQL generation. The optimized general entity recognition module is invoked to perform deep analysis on the preprocessed query text, accurately extracting core business entities such as industry-related object names, data indicators, business terms, and process nodes, ensuring the comprehensiveness and industry adaptability of entity extraction. For business entities with semantic ambiguity or polysemy, a customized masking mechanism is adopted. The ambiguous entity content is replaced by a preset specific mask identifier, which effectively avoids the interference of ambiguous information on the subsequent SQL generation logic and reduces the problem of decreased SQL generation accuracy caused by semantic mapping deviation. The final output is the preprocessed unambiguous query text and the structured set of core business entities, including entity type, original description, mask identifier and other meta-information. This provides semantically clear and structurally standardized high-quality input data for the subsequent Text to SQL algorithm module, helping to improve the accuracy of SQL statement generation and execution effectiveness.

3. The method for general-domain natural language interaction data processing and report generation based on a large model as described in claim 1, characterized in that, The optimization process of the Text to SQL algorithm, which combines RAG retrieval with a general knowledge graph, specifically includes: By calling a general domain-specific entity extraction model, and relying on its multi-scenario adaptability and multi-category entity coverage advantages optimized on the basis of cross-industry corpus pre-training, a second accurate entity extraction is performed on the preprocessed query text, and the entity mask is rewritten for the original query. The RAG retrieval mechanism is activated, and relevant structured data indexes in the general domain database are retrieved based on the extracted core business entities to establish a preliminary connection between the query and the database; Simultaneously, the constructed general business entity knowledge graph is retrieved, and the accurate database association content corresponding to the masked entity is obtained through entity similarity matching, so that the masked entity can be accurately replaced. The rewritten query text, the data index obtained from RAG retrieval, and the entity association content matched by the knowledge graph are input into the large model to generate an executable SQL statement for a general domain database. The accuracy of SQL generation is improved through dual retrieval optimization.

4. The method for general-domain natural language interaction data processing and report generation based on a large model as described in claim 1, characterized in that, The specific implementation process of the structured data-driven report generation algorithm module includes: Receive the execution results of the SQL statement generated by the Text to SQL algorithm module. The results are general-purpose structured data, covering various core business data such as business indicator values, operational record data, and process execution data. We conduct systematic data cleaning on the received structured data to accurately remove outliers and missing values. At the same time, we standardize and unify the format of multi-source heterogeneous data to ensure the data quality and reliability of subsequent analysis processes from the data source. Based on cross-industry general reporting standards and a flexibly configurable template system, we have designed targeted multi-dimensional data parsing rules to perform in-depth processing on cleaned structured data, including core operations such as dimension splitting, indicator calculation, and trend analysis, to fully explore the business value behind the data. It automatically extracts key information from data processing results, including summaries of core indicators, alerts to abnormal data, business trend analysis, and decision-making reference suggestions. The information is organized in an orderly manner according to a standardized report format to generate intuitive and easy-to-understand visual data reports. It supports exporting reports in multiple formats and online previewing to meet the needs of different scenarios. At the same time, it fully preserves the traceability link of the original data, clearly records the entire process of data flow and processing, and provides verifiable and traceable data support for users in various industries.

Images