ChatBI data processing methods, systems, equipment, and media

Abstract

This application relates to the field of data processing technology, and discloses a ChatBI data processing method, system, device, and medium. The ChatBI data processing method includes: acquiring metadata from multi-source heterogeneous data sources, and generating semantically dense vectors and keyword sparse vectors to construct a target vector library, while simultaneously generating the connection configuration required by the database query engine. When a natural language query request is received, semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors are performed in the target vector library to determine the target metadata matching the request. Based on the determined target metadata, a corresponding target database query statement is generated, and the statement is sent to the database query engine with the connection configuration loaded for execution to obtain the query result. This application solves the technical problems of low accuracy in natural language understanding and data query matching, and difficulties in cross-source execution when facing multi-source heterogeneous data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to ChatBI data processing methods, systems, devices, and media. Background Technology

[0002] With the development of big data technology, enterprises and organizations have accumulated massive amounts of business data, which are often scattered across different databases or data warehouses, forming a multi-source heterogeneous data environment. To extract value from this data, users need to query and analyze it. However, existing data query methods often require users to possess professional database language skills (such as SQL) or rely on simple question-and-answer systems that can only handle a single data source. When faced with multi-source heterogeneous data, due to the complexity of data table structures and inconsistent naming conventions, relying solely on traditional keyword matching is insufficient to accurately understand the mapping relationship between user intent and physical table structure. Conversely, relying solely on semantic vector retrieval is prone to errors in proper noun matching, resulting in low accuracy of generated query statements. Furthermore, manually configuring and maintaining connection information for multiple data sources is extremely tedious and error-prone.

[0003] Therefore, there is an urgent need for a ChatBI data processing method to solve the problems of high threshold for cross-source data query, inaccurate semantic understanding, and complex configuration and maintenance in related technologies. Summary of the Invention

[0004] This application provides a ChatBI data processing method, system, device, and medium to solve the problems of high threshold for cross-source data query, inaccurate semantic understanding, and complex configuration and maintenance in related technologies.

[0005] In a first aspect, this application provides a ChatBI data processing method, which includes:

[0006] Obtain metadata from multi-source heterogeneous data sources, generate semantically dense vectors and keyword sparse vectors for the metadata respectively, construct a target vector library, and generate data source connection configurations for the database query engine based on the metadata; In response to a received natural language query request, semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors are performed in the target vector library to determine the target metadata that matches the natural language query request. Based on the target metadata, generate the target database query statement corresponding to the natural language query request; The target database query statement is sent to the database query engine with the data source connection configuration loaded for query execution, and the query results are obtained.

[0007] The ChatBI data processing method provided in this application constructs a vector library by acquiring metadata from multiple heterogeneous data sources and generating semantically dense vectors and keyword sparse vectors respectively. This method simultaneously leverages the generalized matching capability of semantics and the precise matching capability of keywords, significantly improving the accuracy of target metadata retrieval when responding to natural language queries and avoiding matching omissions or errors caused by a single retrieval method. Furthermore, by automatically generating the data source connection configuration for the database query engine based on metadata, it achieves automatic synchronization and adaptation between the query execution layer and the metadata management layer, enabling the generated database query statements to be executed smoothly within a unified engine. This achieves efficient and accurate conversion from natural language to multi-source heterogeneous data query results, improving the data analysis efficiency and user experience of the ChatBI system.

[0008] In one optional implementation, semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors are performed in the target vector library to determine target metadata that matches the natural language query request, including: The first retrieval score is obtained by retrieving semantically dense vectors using the inner product algorithm; The second retrieval score is obtained by retrieving sparse vectors of keywords using the BM25 algorithm. The first search score and the second search score are weighted according to a preset weight ratio to determine the target metadata that matches the natural language query request.

[0009] In one optional implementation, the data source connection configuration for generating the database query engine based on metadata includes: Extract data source connection information from metadata. The data source connection information includes at least one of the following: data source type, connection address, database name, and user information. Deduplication of data source connection information is performed so that the same data source corresponds to only one unique connection configuration in the database query engine; Synchronize the deduplicated data source connection information to the database query engine.

[0010] In one alternative implementation, prior to the steps of performing semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors in the target vector library in response to a received natural language query request, the method further includes: Intent recognition is performed on the original question input by the user to determine the type of the original question; If the original question is a data query request, then check whether the original question is missing any preset key information; If key information is missing, generate rhetorical questions or follow-up questions to interact with the user and complete the key information; Based on the completed key information, the original question is rewritten to obtain a natural language query request.

[0011] In one optional implementation, semantically dense vectors and keyword sparse vectors of metadata are generated, respectively, including: Metadata is converted into text description information according to a preset natural language description template. The text description information includes at least one of the following: table name, field name, field type, and Chinese definition. The text description information is encoded to generate semantically dense vectors with a preset dimension and sparse vectors based on keywords.

[0012] In one optional implementation, based on the target metadata, a target database query statement corresponding to the natural language query request is generated, including: The difficulty level of natural language query requests is classified using a pre-defined text classification model to obtain the classification results; Based on the classification results, natural language query requests are sent to different pre-defined large language models for processing.

[0013] In one optional implementation, after sending the target database query statement to a database query engine with the data source connection configuration loaded for query execution, the method further includes: If the query fails, retrieve the error message returned by the database query engine; Based on the target database query statement, error messages, and target metadata, the statement is corrected to generate a corrected target database query statement. The corrected target database query statement is resent to the database query engine for execution. The correction and resending process is repeated up to a preset number of times.

[0014] Secondly, this application provides a ChatBI data processing system, which includes a target vector library, a database query engine, and a processing module; The target vector library is used to store semantically dense vectors and keyword sparse vectors of metadata from multi-source heterogeneous data sources. The database query engine is configured with data source connection settings based on metadata, which connect to multiple heterogeneous data sources for cross-source query operations. The processing module is communicatively connected to the target vector library and the database query engine, respectively, and is used to execute the ChatBI data processing method of the first aspect or any of its corresponding embodiments, so as to recall target metadata through the target vector library and generate target database query statements when a natural language query request is received, and control the database query engine to execute the query and return the results.

[0015] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the ChatBI data processing method of the first aspect or any corresponding embodiment described above.

[0016] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the ChatBI data processing method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic flowchart of a first type of ChatBI data processing method according to an embodiment of this application; Figure 3 This is a schematic diagram of a second flowchart of the ChatBI data processing method according to an embodiment of this application; Figure 4 This is a schematic diagram of the data processing flow of the ChatBI system according to an embodiment of this application; Figure 5 This is a structural block diagram of the ChatBI data processing apparatus according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained.

[0022] ChatBI refers to a business intelligence system that enables data querying, analysis, and visualization through natural language dialogue (Chat).

[0023] Multi-source heterogeneous data: refers to a collection of data with different sources (such as MySQL, Oracle, Hive, etc.) and different structures (relational, non-relational, semi-structured).

[0024] Metadata: Data that describes data, such as structured information like database names, table names, field names, field types, field comments, and table comments.

[0025] Schema: The schema definition of a database, which typically contains a set of tables and their relationships.

[0026] Embedding: A technique for mapping discrete variables (such as text) to continuous vectors.

[0027] Semantically dense vectors: Floating-point vectors (e.g., 1024-dimensional) generated by deep learning models can capture deep semantic relationships in text (e.g., the similarity between revenue and income).

[0028] Keyword sparse vector: A vector generated based on word frequency statistics, focusing on precise literal matching, with most elements in the vector being 0.

[0029] HNSW (Hierarchical Navigable Small World): An efficient graph indexing algorithm for approximate nearest neighbor search, used to accelerate vector retrieval.

[0030] BM25 (Best Matching 25): A weighted algorithm based on a probabilistic retrieval model, often used to evaluate the relevance of search terms to documents, primarily for keyword retrieval.

[0031] Trino: A high-performance distributed SQL query engine (formerly PrestoSQL) that supports federated queries, which allows querying multiple data sources through a single interface.

[0032] Catalog: In Trino, this is the top-level namespace used to define data source connection configurations, containing information such as connection address and user credentials.

[0033] LLM (Large Language Model): A deep learning model with a large number of parameters and trained on a large corpus (such as Qwen3).

[0034] Prompt: A text instruction given to a large language model to guide it in generating specific outputs.

[0035] CoT (Chain of Thought): A cue word engineering technique that guides models to perform step-by-step logical reasoning before giving the final answer.

[0036] GPTQ-4bit: A model quantization technique that compresses model parameters to 4 bits to reduce memory usage and accelerate inference, enabling it to run on consumer-grade graphics cards.

[0037] FastText: A lightweight text classification model with extremely fast computation speed, often used for intent recognition and text classification.

[0038] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, the ChatBI system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0039] Specifically, the terminal device can be a smartphone, tablet, laptop, desktop computer, etc., through which the user inputs a natural language query request. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranets, local area networks, wide area networks, mobile communication networks, and combinations thereof, used to realize data transmission between the terminal device and the server.

[0040] In related technologies, when users face massive heterogeneous data scattered across different systems, they often need to manually look up table structures and write complex SQL statements. In response to the technical problems faced by users when performing cross-source data queries, such as high learning costs, difficulty in finding metadata, and low accuracy of query statements generated by existing automatic conversion tools, this disclosure provides a ChatBI data processing method to achieve the technical effect of enabling users to accurately and efficiently obtain multi-source heterogeneous data analysis results using only natural language.

[0041] According to an embodiment of this application, a ChatBI data processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] This embodiment provides a ChatBI data processing method, which can be used in the aforementioned terminal devices, such as desktop computers, laptops, etc. Figure 2 This is a flowchart of the ChatBI data processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain metadata from multi-source heterogeneous data sources, generate semantically dense vectors and keyword sparse vectors of the metadata respectively to construct a target vector library, and generate the data source connection configuration of the database query engine based on the metadata.

[0043] Multi-source heterogeneous data sources refer to data storage systems with different origins and structures. For example, they can be a combination of relational databases (such as MySQL and Oracle), non-relational databases (such as MongoDB), or data warehouses (such as Hive). These data sources store specific business data.

[0044] Metadata is data that describes data, such as the table name, column name, field type, and field comments.

[0045] Semantically dense vectors are vectors that transform text into high-dimensional real-valued vectors using deep learning models (such as Embedding models). These vectors capture the deep semantic information of the text. For example, "revenue" and "sales revenue" are literal words, but they are very close in semantically dense vector space. "Keyword sparse vectors," on the other hand, are vectors generated based on word frequency statistics (such as the BM25 algorithm), focusing on precise literal matching. For example, for a specific product code "SKU_2023," sparse vectors ensure that it is accurately retrieved without being confused with semantically similar but completely different words.

[0046] The specific steps of this process are as follows: First, scan and extract the metadata from each data source. Then, for each piece of metadata (such as a table description), generate a dense vector using a semantic model and a sparse vector using a keyword statistics algorithm. Store these two types of vectors in the target vector library. Simultaneously, parse the connection information (such as IP address, port, and database name) from the metadata to automatically generate a connection configuration adapted to the database query engine, ensuring that the engine has the necessary permissions and access to these data sources.

[0047] For example, suppose a company has a MySQL database storing "user information" and a Hive data warehouse storing "transaction records." First, both data sources are automatically scanned to extract the structural information (metadata) of the "user_info" and "transaction_log" tables. Next, this metadata is processed: on one hand, the comment "stores basic customer information" in the "user_info" table is interpreted to generate a semantically dense vector; on the other hand, the table name "user_info" and the field name "uid" are extracted as keywords to generate a keyword sparse vector. These two types of vectors are stored in a dedicated vector database (building the target vector library). Simultaneously, based on the extracted MySQL and Hive connection information (IP, account, etc.), a "Catalog" or connection configuration is automatically created in the background query engine, enabling the query engine to directly access these two databases.

[0048] Step S202: In response to the received natural language query request, perform semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors in the target vector library to determine the target metadata that matches the natural language query request.

[0049] Natural language query requests are queries that users input in everyday language, such as "Please show me the total sales of Region A last year".

[0050] Upon receiving the request, it is vectorized, generating corresponding dense and sparse query vectors. Then, both types of searches are performed in parallel within the target vector library: Semantic retrieval refers to finding metadata that is semantically closest to the query intent. For example, if a user says "profit situation," semantic retrieval can find metadata related to the profit and loss statement.

[0051] Keyword search refers to finding metadata that exactly matches the query keywords. For example, if a user mentions the abbreviation of a specific table name, "T_Sales," keyword search can accurately locate that table.

[0052] By combining the results of these two searches (e.g., through weighted scoring), the system filters out the portion of metadata that best matches the user's needs from a massive amount of metadata—the target metadata. This ensures that the ChatBI system can both understand the user's vague intent and accurately target specialized terminology.

[0053] For example, a user enters a question on the interface: "Please check last month's sales figures." Upon receiving this request, the system performs two operations simultaneously in the vector library: Semantic retrieval: Analyzing the semantics of "sales revenue" reveals that it is very close to the semantic vector of the "transaction_log" table (annotated as: transaction log) in the vector library.

[0054] Keyword retrieval: If the user enters "search for data in table_a", the system will use sparse vectors to precisely match the table named "table_a". The results from both searches are then comprehensively evaluated (e.g., weighted sorting) to ultimately determine that the "transaction_log" table and its "amount" field are what the user needs; this information constitutes the target metadata.

[0055] Step S203: Based on the target metadata, generate the target database query statement corresponding to the natural language query request.

[0056] The target database query statement usually refers to a Structured Query Language statement, such as an SQL query statement.

[0057] The user's natural language query request is combined with the target metadata filtered in step S202 and passed as input to the generation module (e.g., a rule-based generator or a large language model). The generation module understands the user's intent (what metric, time range, and filtering conditions to query) and uses the accurate table and field names provided in the target metadata to construct a grammatically correct query statement. For example, the total sales amount of region A last year can be converted to `SELECT sum(amount) FROM sales_table WHERE region='A' AND year='2024'`.

[0058] Step S204: Send the target database query statement to the database query engine that has been loaded with the data source connection configuration for query execution and obtain the query results.

[0059] The database query engine is a component responsible for executing query logic. It has pre-loaded the data source connection configuration generated in step S201, and therefore has the ability to connect to underlying multi-source heterogeneous data sources.

[0060] The generated SQL statement is sent to the engine. The engine parses the statement, connects to the specific physical database (such as MySQL or Hive) using the connection configuration, issues the query command, and retrieves the data returned by the underlying data source. Finally, the ChatBI system returns the data obtained from this execution as the query results to the user.

[0061] For example, the ChatBI system sends the SQL statement generated in step S203 to the backend database query engine. The engine recognizes that it needs to query the `transaction_log` table. Based on the pre-generated configuration in S201, it knows that this table is located in the Hive data source. Therefore, the engine automatically connects to Hive, performs aggregation calculations, and returns the final calculated number (e.g., 1,000,000) as the query result to the system. The ChatBI system then displays this result to the user, completing a full data query-answering process.

[0062] In summary, the ChatBI data processing method provided in this embodiment constructs a vector library by acquiring metadata from multiple heterogeneous data sources and generating semantically dense vectors and keyword sparse vectors respectively. This method simultaneously leverages the generalized matching capability of semantics and the precise matching capability of keywords, significantly improving the accuracy of target metadata retrieval when responding to natural language queries and avoiding matching omissions or errors caused by a single retrieval method. Furthermore, by automatically generating the data source connection configuration for the database query engine based on metadata, it achieves automatic synchronization and adaptation between the query execution layer and the metadata management layer, enabling the generated database query statements to be executed smoothly within a unified engine. This achieves efficient and accurate conversion from natural language to multi-source heterogeneous data query results, improving the data analysis efficiency and user experience of the ChatBI system.

[0063] This embodiment provides a ChatBI data processing method, which can be used in the aforementioned terminal devices, such as desktop computers, laptops, etc. Figure 3 This is a flowchart of the ChatBI data processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain metadata from multi-source heterogeneous data sources, generate semantically dense vectors and keyword sparse vectors of the metadata respectively to construct a target vector library, and generate the data source connection configuration of the database query engine based on the metadata.

[0064] To better understand this metadata, it is converted into text descriptions according to a predefined natural language template. For example, the table name t_sale, the field amt, and the comment "sales" are converted into the text: "Data table t_sale, containing the field amt, representing sales data." This text is then encoded: on one hand, an embedding model is used to generate a semantically dense vector (with a predefined dimension of 1024) that captures deeper meanings (such as the relationship between "sales" and "revenue"); on the other hand, a statistical model is used to generate sparse keyword vectors that focus on literal matching. Both types of vectors are stored in a target vector library.

[0065] Simultaneously, physical connection information (such as IP address, port, and database name) of the data source is extracted from the metadata. Given that multiple business tables may originate from the same physical database, this information is deduplicated to ensure that only one unique connection configuration (Catalog) is generated for the same physical data source. This configuration is then synchronized to the database query engine (such as Trino) to enable cross-source execution capabilities.

[0066] Step S302: In response to the received natural language query request, perform semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors in the target vector library to determine the target metadata that matches the natural language query request.

[0067] Before performing the search, to ensure the validity of the query, according to claim 4, the original question input by the user is first processed, as follows: Step S3021: Perform intent recognition on the original question input by the user to determine the type of the original question.

[0068] The initial question refers to the initial text or speech-transcribed text entered by the user in the ChatBI interface. It may contain unstructured spoken expressions, small talk, or incomplete query requests. For example, a user might enter "Hello," "The data above is incorrect," or "Check the sales figures."

[0069] Intent recognition refers to the process of semantically analyzing input text and classifying it into several predefined types (such as data query, casual conversation, operation instructions, etc.).

[0070] Type refers to the category label to which the question belongs.

[0071] For example, if a user enters "The weather is really nice," the ChatBI system identifies it as a "casual chat" request and directly calls the casual chat API to reply, without proceeding to the subsequent data processing flow. Conversely, if a user enters "Please check last quarter's profit and loss statement," the ChatBI system identifies it as a "data query request" and triggers the subsequent inspection process. Through this step, the ChatBI system can focus its computing resources more on actual business analysis needs.

[0072] Step S3022: If the type of the original question is a data query request, check whether the original question is missing any preset key information.

[0073] Key information refers to the elements necessary to complete an effective data query, which typically include time dimensions (such as "2023"), indicator objects (such as "sales revenue"), and statistical dimensions (such as "by branch office").

[0074] If the original question is determined to be a data query request, the language model will be used to scan the question in conjunction with the context. It will then determine whether the original question is sufficient to constitute a complete SQL query condition.

[0075] For example, a user enters "view sales". During the check, the ChatBI system finds that although the indicator "sales" is present, the key information of the preset "time range" is missing (i.e., the system does not know which day's sales the user wants to view), and therefore judges it as "key information missing".

[0076] In step S3023, if key information is missing, generate rhetorical questions or follow-up questions to interact with the user and complete the key information.

[0077] A rhetorical question or follow-up question is a prompting phrase generated by the ChatBI system, designed to elicit missing information from the user.

[0078] Interaction refers to the multi-round dialogue process of the ChatBI system asking questions and the user answering them.

[0079] Completion refers to filling in previously blank information slots by obtaining new input from the user.

[0080] When the ChatBI system detects missing critical information, it will not force the generation of incorrect SQL. Instead, it will use a large model to output targeted rhetorical questions and engage in multiple rounds of dialogue with the user.

[0081] For example, in response to the "view sales figures" request above, the ChatBI system generates a follow-up question: "Which time period would you like to query for sales figures? For example, last year or this month?" After seeing the follow-up question, the user replies: "Last month." At this point, the ChatBI system will capture the information "last month," thus completing the crucial information of the "time range." This mechanism effectively reduces the query failure rate caused by users' unfamiliarity with the business or incomplete expression.

[0082] Step S3024: Based on the completed key information, the original question is rewritten to obtain the natural language query request.

[0083] Rewriting refers to the process of using natural language generation technology to merge the original ambiguous problem with supplementary information from the context and reconstruct it into a semantically complete and unambiguous independent sentence.

[0084] A natural language query request refers to the final query text that has been rewritten to meet all the necessary conditions for vector retrieval and SQL generation.

[0085] By using the completed key information (such as "last month") and the user's original question (such as "view sales"), and combining the context of the dialogue, the question can be rewritten.

[0086] For example, the ChatBI system combines "view sales" and "last month" with the current time (let's say October 2023) to rewrite the question into a standard natural language query: "Query the sales volume of all products in September 2023." This rewritten sentence contains clear time constraints and metric definitions, and is then sent to a target vector library for mixed retrieval, ensuring that the retrieved metadata and the generated SQL statement are highly accurate.

[0087] Subsequently, a hybrid retrieval was performed on the request. Two retrieval methods were executed in parallel within the vector library: The similarity between the query vector and the semantically dense metadata vector is calculated using the inner product algorithm to obtain the first retrieval score (focusing on semantic matching).

[0088] The BM25 algorithm is used to calculate the matching degree between the query and the sparse vector of metadata keywords, resulting in a second retrieval score (emphasizing literal exact matching). The two scores are then weighted and summed according to a preset weight (e.g., 0.4:1), and the table structure information with the highest scores is selected as the target metadata.

[0089] Step S303: Based on the target metadata, generate the target database query statement corresponding to the natural language query request.

[0090] In this step, the ChatBI system uses a large language model to convert natural language into SQL statements. To optimize resources, the ChatBI system first uses a lightweight text classification model (such as FastText) to categorize queries by difficulty: Simple (e.g., single-table query): Send to a model with fewer parameters and faster response.

[0091] Complex cases (such as multi-table joins and nested queries): These are sent to a model with a large number of parameters and strong reasoning capabilities for processing. The model combines the found target metadata with the user's query request to generate the final target database query statement.

[0092] Step S304: Send the target database query statement to the database query engine that has been loaded with the data source connection configuration for query execution and obtain the query results.

[0093] The ChatBI system sends the generated SQL to the query engine that has been configured with the connection in step S301 for execution. However, errors may occur during execution (such as misspelled field names).

[0094] If the query engine returns an execution failure and specific error message.

[0095] The system will re-input the error message, the erroneous SQL, and the target metadata into the large model, requiring the model to correct the statements.

[0096] The revised SQL will be resent for execution. This process will repeat a maximum of a preset number of times (e.g., 3 times) until execution succeeds or the maximum number of attempts is reached. Finally, the successfully executed query results will be returned and displayed to the user.

[0097] In summary, the ChatBI data processing method provided in this embodiment constructs an end-to-end intelligent multi-source heterogeneous data query system. First, by employing hybrid vector retrieval, it cleverly combines the advantages of semantic understanding and keyword matching, solving the retrieval challenges caused by non-standard table structure naming and semantic ambiguity in multi-source heterogeneous data, laying a solid foundation for accurate SQL generation. Second, through a pre-emptive interactive intent refinement mechanism and a post-emptive automatic error correction and retry mechanism, a dual guarantee of pre-emptive guidance and post-event remediation is formed, greatly improving the query success rate in complex business scenarios. Simultaneously, with automated connection configuration synchronization and a hierarchical processing strategy that adapts resources, it not only achieves seamless execution of cross-source heterogeneous data but also effectively reduces the system's operational complexity and computational costs. This effectively solves the core technical problems of existing technologies, such as high barriers to entry for natural language data retrieval, low accuracy, difficulty in cross-source execution, and high resource consumption, achieving an efficient, accurate, and economical intelligent data analysis experience.

[0098] To better illustrate the ChatBI data processing method of the embodiments, a preferred embodiment will be provided below. This embodiment is intended to describe the implementation process of this disclosure in detail, but is not intended to limit the scope of protection of this disclosure.

[0099] This embodiment provides a ChatBI system and its data processing method. The system mainly consists of a target vector library, a database query engine (Trino), and processing modules. The system aims to solve the problems of difficult querying of multi-source heterogeneous data, cumbersome configuration, and low accuracy through an intelligent process. The data processing flow of the system is as follows: Figure 4 As shown, it specifically includes: The first step is system initialization and infrastructure construction.

[0100] At the beginning of system operation, the processing module first performs the work of accessing and preparing metadata.

[0101] The system is equipped with a dedicated data exploration tool that can proactively connect to various heterogeneous data sources, automatically explore and obtain information such as database table structure and semi-structured data attribute descriptions. Simultaneously, the system also supports manual metadata entry to supplement information that is difficult for machines to access, such as business definitions, thereby establishing a complete metadata asset.

[0102] The processing module organizes the acquired metadata according to a preset first text format. This first text format combines the database name, table name (including Chinese definitions), and field name (including Chinese definitions and types) into a natural language description. For example: "Database x is a schema, which contains a table xx (table's Chinese name is zxx)". Subsequently, the system uses a hybrid embedding model to process this text, generating a 1024-dimensional semantically dense vector and a keyword sparse vector, respectively. An example of the first text prompt word template is shown below: .

[0103] For semantically dense vectors, the system builds an HNSW index to support millisecond-level semantic retrieval.

[0104] For sparse vectors of keywords, the system stores them in a database to support precise matching.

[0105] Meanwhile, the raw JSON content of the metadata is stored in the target vector library in scalar form for subsequent precise filtering.

[0106] This system has undergone significant modifications to the open-source Trino engine, specifically the trino-main module. After metadata is generated, the processing module extracts data source connection information (such as data source type, IP address, database name, username, etc.) from it. To avoid resource waste, the system employs a deduplication generation strategy: when multiple metadata points to the same physical data source, the system merges them, generating only a unique Catalog configuration in Trino.

[0107] Next is the reception and understanding of natural language queries.

[0108] Once the infrastructure is ready, users can initiate queries via text or voice input through the interface. The processing module handles requests according to the following process: The system first calls the language model to identify the user's intent. If it determines that the user's question does not belong to data query or analysis / mining (e.g., casual conversation), it directly rejects the question or switches to casual conversation mode to avoid wasting computing resources.

[0109] If the request is confirmed to be a data query, the large model will check the context to see if any key information (such as the target object or time range) is missing. For example, if a user asks "How are the sales?", the model will ask, "Which time period's sales would you like to query?". Through this series of follow-up questions, the system guides the user to complete the information and eliminate ambiguity.

[0110] Using the completed information and historical context, the system rewrites the user's original question into a separate, complete, and unambiguous natural language query request.

[0111] This is followed by hybrid retrieval and metadata recall.

[0112] The processing module performs a dual hybrid search on the rewritten question in the target vector library: The similarity of semantically dense vectors is calculated using the Inner Product algorithm.

[0113] The BM25 algorithm is used to calculate the matching degree of the keyword sparse vector.

[0114] The system weights and merges the two according to a pre-set weight ratio (dense:sparse, such as 0.4:1). This ratio has been optimized to reflect that in precise data queries, literal matching (such as accurate table names and SKU codes) is more important than fuzzy semantics.

[0115] If the user explicitly specifies the query scope (such as querying within the financial database), the system will use a scalar index for filtering.

[0116] Through the above steps, the system can accurately retrieve the top 5 most matching metadata from massive amounts of metadata, thus achieving pattern connection.

[0117] Then comes hierarchical reasoning and SQL generation.

[0118] To reduce computing costs while maintaining high accuracy, the system adopts a tiered processing strategy: The FastText model is used to categorize user questions into three levels: Easy: If it involves single-table queries or simple aggregation (Group By), then it enters the single-expert link.

[0119] Medium: If single-table functions or simple multi-table joins are involved, then the process enters the single-expert review process.

[0120] Hard: If it involves complex multi-table joins, window functions, etc., then it will enter the multi-expert review process.

[0121] The system will retrieve the top 5 metadata, the current time, and high-quality historical question-and-answer pairs, and then input them into a pre-defined second text prompt template. This second text prompt template contains... <think>Labels force the model to follow the Chain of Thought (CoT) process: first describe the problem, then select a table, next choose fields, and finally devise the Where conditions and check for compliance with rules such as only_full_group_by. An example of the second text prompt template is shown below: .

[0122] The expert model used in this system is a finely tuned Qwen3-14B large language model, which has undergone GPTQ-4-bit quantization. In the review process, the system introduces a reinforcement learning-trained model to optimize the selection from multiple generated SQL candidates.

[0123] Finally, there is the closed loop of cross-source execution and automatic error correction.

[0124] After generating the target database query statement (SQL), the execution phase begins: The system performs simple rule-based syntax corrections on the SQL and enforces security restrictions such as LIMIT to prevent queries from overloading the database.

[0125] The revised SQL is sent to the database query engine (Trino). Since Trino has loaded the full Catalog configuration, it can directly execute cross-source join queries (such as joining MySQL tables with Hive tables), shielding it from the underlying heterogeneity.

[0126] If Trino encounters an error (such as an incorrect field name), the system will not directly report the error to the user. The processing module will capture the error information and populate it, along with the original SQL and metadata, into a pre-defined third-party text prompt template, requesting the large model to make targeted corrections. This process will be repeated up to three times. This mechanism effectively addresses subtle errors caused by model illusions. An example of a third-party text prompt template is shown below: .

[0127] Upon successful execution, the result is returned in JSON format. The system uses a custom Markdown component library to automatically render it into a table or chart based on the data type, and then presents it to the user. If the process fails after multiple retries, a fallback answer is generated to guide the user to adjust their question.

[0128] Through the above implementation methods, the ChatBI system in this embodiment improves the accuracy of SQL generation and has extremely high adaptability.

[0129] This embodiment also provides a ChatBI data processing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0130] This embodiment provides a ChatBI data processing device, such as... Figure 5 As shown, it includes: Module 501 is used to obtain metadata from multi-source heterogeneous data sources, generate semantically dense vectors and keyword sparse vectors of the metadata respectively, to build a target vector library, and generate data source connection configuration for the database query engine based on the metadata. The retrieval module 502 is used to respond to the received natural language query request by performing semantic retrieval based on semantically dense vectors and keyword retrieval based on keyword sparse vectors in the target vector library to determine the target metadata that matches the natural language query request. The generation module 503 is used to generate the target database query statement corresponding to the natural language query request based on the target metadata; The execution module 504 is used to send the target database query statement to the database query engine that has been loaded with the data source connection configuration for query execution and to obtain the query results.

[0131] In one alternative implementation, the retrieval module 502 is used for: The first retrieval score is obtained by retrieving semantically dense vectors using the inner product algorithm; The second retrieval score is obtained by retrieving sparse vectors of keywords using the BM25 algorithm. The first search score and the second search score are weighted according to a preset weight ratio to determine the target metadata that matches the natural language query request.

[0132] In one alternative implementation, the construction module 501 is configured to: Extract data source connection information from metadata. The data source connection information includes at least one of the following: data source type, connection address, database name, and user information. Deduplication of data source connection information is performed so that the same data source corresponds to only one unique connection configuration in the database query engine; Synchronize the deduplicated data source connection information to the database query engine.

[0133] In an optional implementation, the retrieval module 502 is further configured to: Intent recognition is performed on the original question input by the user to determine the type of the original question; If the original question is a data query request, then check whether the original question is missing any preset key information; If key information is missing, generate rhetorical questions or follow-up questions to interact with the user and complete the key information; Based on the completed key information, the original question is rewritten to obtain a natural language query request.

[0134] In one alternative implementation, the construction module 501 is configured to: Metadata is converted into text description information according to a preset natural language description template. The text description information includes at least one of the following: table name, field name, field type, and Chinese definition. The text description information is encoded to generate semantically dense vectors with a preset dimension and sparse vectors based on keywords.

[0135] In one alternative implementation, the generation module 503 is used for: The difficulty level of natural language query requests is classified using a pre-defined text classification model to obtain the classification results; Based on the classification results, natural language query requests are sent to different pre-defined large language models for processing.

[0136] In an optional implementation, execution module 504 is further configured to: If the query fails, retrieve the error message returned by the database query engine; Based on the target database query statement, error messages, and target metadata, the statement is corrected to generate a corrected target database query statement. The corrected target database query statement is resent to the database query engine for execution. The correction and resending process is repeated up to a preset number of times.

[0137] The ChatBI data processing apparatus provided in this application can execute the ChatBI data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0138] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0139] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural schematic for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0140] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0141] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the ChatBI data processing method of embodiments of this application.

[0142] Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0143] This application also provides a computer-readable storage medium. The methods described above according to this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the ChatBI data processing method shown in the above embodiments is implemented.

[0144] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0145] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.< / think>

Claims

1. A ChatBI data processing method, characterized in that, The method includes: Obtain metadata from multi-source heterogeneous data sources, generate semantically dense vectors and keyword sparse vectors of the metadata respectively to construct a target vector library, and generate data source connection configuration for the database query engine based on the metadata; In response to a received natural language query request, semantic retrieval based on the semantically dense vector and keyword retrieval based on the keyword sparse vector are performed in the target vector database to determine the target metadata that matches the natural language query request. Based on the target metadata, generate the target database query statement corresponding to the natural language query request; The target database query statement is sent to the database query engine that has been loaded with the data source connection configuration for query execution, and the query result is obtained.

2. The method according to claim 1, characterized in that, The step of performing semantic retrieval based on the semantically dense vectors and keyword retrieval based on the keyword sparse vectors in the target vector database to determine the target metadata matching the natural language query request includes: The semantically dense vector is retrieved using the inner product algorithm to obtain the first retrieval score; The sparse vector of the keyword is retrieved using the BM25 algorithm to obtain the second retrieval score; The first search score and the second search score are weighted according to a preset weight ratio to determine the target metadata that matches the natural language query request.

3. The method according to claim 1, characterized in that, The data source connection configuration for generating the database query engine based on the metadata includes: Extract data source connection information from the metadata, wherein the data source connection information includes at least one of data source type, connection address, database name, and user information; The data source connection information is deduplicated so that the same data source corresponds to only one unique connection configuration in the database query engine; The deduplicated data source connection information is synchronized to the database query engine.

4. The method according to claim 1, characterized in that, Before the steps of performing semantic retrieval based on the semantically dense vector and keyword retrieval based on the keyword sparse vector in response to a received natural language query request, the method further includes: Intent recognition is performed on the original question input by the user to determine the type of the original question; If the type of the original question is a data query request, then check whether the original question is missing any preset key information; If the key information is missing, generate rhetorical questions or follow-up questions to interact with the user and complete the key information; Based on the completed key information, the original question is rewritten to obtain the natural language query request.

5. The method according to claim 1, characterized in that, The generation of the semantically dense vector and the keyword sparse vector of the metadata respectively includes: The metadata is converted into text description information according to a preset natural language description template. The text description information includes at least one of the following: table name, field name, field type, and Chinese definition. The text description information is encoded to generate a semantically dense vector with a preset dimension and a sparse vector based on keywords.

6. The method according to claim 1, characterized in that, The step of generating the target database query statement corresponding to the natural language query request based on the target metadata includes: The natural language query requests are classified into difficulty levels using a preset text classification model to obtain classification results; Based on the classification results, the natural language query request is sent to different preset large language models for processing.

7. The method according to claim 1, characterized in that, After sending the target database query statement to the database query engine that has been loaded with the data source connection configuration for query execution, the process further includes: If the query fails, obtain the error information returned by the database query engine; Based on the target database query statement, the error message, and the target metadata, the statement is corrected to generate a corrected target database query statement. The corrected target database query statement is resent to the database query engine for execution, and the correction and resending process is repeated up to a preset number of times.

8. A ChatBI data processing system, characterized in that, The system includes a target vector library, a database query engine, and a processing module; The target vector library is used to store semantically dense vectors and keyword sparse vectors of metadata from multi-source heterogeneous data sources. The database query engine is configured with a data source connection configuration generated based on the metadata, and connects to the multi-source heterogeneous data source respectively for cross-source query operations. The processing module is communicatively connected to the target vector library and the database query engine, respectively, and is used to execute the ChatBI data processing method as described in any one of claims 1 to 7, so that when a natural language query request is received, the module retrieves target metadata through the target vector library and generates a target database query statement, controls the database query engine to execute the query and return the result.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the ChatBI data processing method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the ChatBI data processing method according to any one of claims 1 to 7.

Images