Data query processing method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence and databases, provides a data query processing method and device, equipment, a storage medium and a program product, and can reduce the use threshold of data query. The method comprises the following steps: inputting a data query question into a large language model to indicate the large language model to extract question elements; according to the questioning elements, determining a candidate business object logic view; according to the metadata of the candidate business object logic view and the data query problem, obtaining a query statement generation cue word; inputting the query statement generation prompt word into the large language model to indicate the large language model to generate a standardized query statement matched with the database grammar type carried by the metadata; and sending the standardized query statement to a unified query layer to indicate the unified query layer to translate the standardized query statement, and routing the translated standardized query statement to a physical data source corresponding to the database grammar type for execution to obtain a data query result.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and database technology, and in particular to a data query processing method, apparatus, computer equipment, storage medium and computer program product. Background Technology

[0002] Physical data sources can store data as physical data tables; they can also utilize heterogeneous physical data sources. Subsequent data queries can then be performed within these physical data sources to retrieve the desired data.

[0003] Data querying in related technologies generally relies on business personnel manually writing data query statements, which places high demands on business personnel and has a high technical threshold. Summary of the Invention

[0004] Therefore, it is necessary to provide a data query and processing method, apparatus, computer equipment, storage medium, and computer program product to address the aforementioned technical problems.

[0005] This application provides a data query and processing method, the method comprising:

[0006] The user's data query question is input into the large language model; the data query question is used to instruct the large language model to extract the query elements from the data query question;

[0007] Based on the question elements, candidate business object logical views are determined from a number of pre-constructed business object logical views; the business object logical views are obtained by semantically abstracting a number of physical data tables based on business logic; the physical data tables are data tables stored in physical data sources.

[0008] Based on the metadata of the logical view of the candidate business object and the data query question, query statement prompts are generated.

[0009] The query statement is used to generate prompt words, which are then input into the large language model. These prompt words instruct the large language model to generate standardized query statements that match the database syntax type carried by the metadata.

[0010] The standardized query statement is sent to the unified query layer; the standardized query statement is used to instruct the unified query layer to translate the standardized query statement, and to route the translated standardized query statement to the physical data source corresponding to the database syntax type for execution, so as to obtain the data query result corresponding to the data query question.

[0011] This application provides a data query processing apparatus, the apparatus comprising:

[0012] The question input module is used to input the user's data query question into the large language model; the data query question is used to instruct the large language model to extract the query elements from the data query question;

[0013] The view determination module is used to determine candidate business object logical views from a number of pre-built business object logical views based on the query elements; the business object logical views are obtained by semantically abstracting a number of physical data tables based on business logic; the physical data tables are data tables stored in physical data sources.

[0014] The prompt word determination module is used to generate prompt words based on the metadata of the logical view of the candidate business object and the data query question;

[0015] The prompt word input module is used to input the prompt words generated by the query statement into the large language model; the prompt words generated by the query statement are used to instruct the large language model to generate a standardized query statement that matches the database syntax type carried by the metadata;

[0016] The statement sending module is used to send the standardized query statement to the unified query layer; the standardized query statement is used to instruct the unified query layer to translate the standardized query statement, route the translated standardized query statement to the physical data source corresponding to the database syntax type for execution, and obtain the data query result corresponding to the data query question.

[0017] This application provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor executes the above-described method.

[0018] This application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor using the methods described above.

[0019] This application provides a computer program product having a computer program stored thereon, the computer program being executed by a processor using the above-described method.

[0020] The aforementioned data query processing method, apparatus, computer equipment, storage medium, and computer program product input the user's data query question into a large language model; the data query question instructs the large language model to extract query elements from the data query question; based on the query elements, candidate business object logical views are determined from several pre-built business object logical views; the business object logical views are obtained by semantically abstracting several physical data tables based on business logic; the physical data tables are data tables stored in physical data sources; based on the metadata of the candidate business object logical views and the data query question, query statement generation prompts are obtained; the query statement generation prompts are input into the large language model; the query statement generation prompts instruct the large language model to generate a standardized query statement that matches the database syntax type carried by the metadata; the standardized query statement is sent to the unified query layer; the standardized query statement instructs the unified query layer to translate the standardized query statement, and routes the translated standardized query statement to the physical data source corresponding to the database syntax type for execution, thereby obtaining the data query result corresponding to the data query question. In the solution provided in this application, users can provide data query questions in natural language, and the corresponding data query results can be automatically and accurately returned to the user, lowering the barrier to entry for data querying. Furthermore, leveraging the natural language processing capabilities of a large language model, this application can automatically extract relatively accurate query elements from the data query questions without manual intervention. Based on business logic, semantic abstraction is performed on several physical data tables to obtain logical views of business objects. The establishment of these logical views improves the generation of multi-level nested subqueries or query statements involving complex multi-table joins (such as SQL), enhancing complex query capabilities. Moreover, based on the query elements, candidate logical views of business objects can be determined from several pre-built logical views. Based on the metadata of the candidate logical views and the data query questions, query statement generation prompts can be obtained. These prompts are then input into the large language model, and leveraging its natural language understanding capabilities, the accuracy of combined queries using aggregate functions (such as COUNT, SUM, AVG) and GROUP BY clauses can be improved, ensuring the accuracy of sorting constraints (such as ORDER). The correct generation of BY and LIMIT can enhance the ability to perform complex queries. In addition, after the large language model generates standardized query statements that match the database syntax type carried by the metadata, the unified query layer can translate them and route them to the corresponding physical data source for execution. This allows the large language model to reduce the number of join operations on physical data tables during the data query statement generation stage, thereby improving accuracy. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a data query processing method in one embodiment;

[0023] Figure 2 This is a schematic diagram of a dual-tower model structure in one embodiment;

[0024] Figure 3 This is another flowchart illustrating the data query processing method in one embodiment;

[0025] Figure 4 This is a structural block diagram of a data query processing device in one embodiment;

[0026] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various objects, but these objects are not limited by these terms. These terms are only used to distinguish the first object from the second object. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions, or any combination of multiple solutions.

[0029] The data query and processing method provided in this application can be executed by a computer device and may include... Figure 1 The steps are shown.

[0030] Step S101: Input the user's data query question into the large language model.

[0031] Users can provide data query questions in natural language; these questions can be called natural language query questions. After obtaining the data query questions, they can be input into the large language model. The data query questions can then be used to instruct the large language model to extract the query elements from them.

[0032] For example, a user's data query question could be "the total population of region A last year". The task of a large language model can be to identify and extract the query elements (core words) from the data query question, and also to classify the query elements by entity type. For example, the entity type of "region A" could be "city", the entity type of "last year" could be "year", and the entity type of "total population" could be "population indicator". Through a large language model, the core semantics of the user's query can be captured relatively accurately.

[0033] Step S102: Based on the query elements, determine the candidate business object logical view from several pre-built business object logical views.

[0034] A physical data source can also be called a database, and the data tables stored in a physical data source can be called physical data tables. Database syntax types may differ between heterogeneous physical data sources.

[0035] The solution proposed in this application may include a logical semantic abstraction layer. The goal of this layer can be to abstract complex, low-level physical data table structures and their relationships into a business-friendly logical view (which can be called a business object logical view). In other words, the business object logical view is obtained by semantically abstracting several physical data tables based on business logic. This simplifies the subsequent query generation process. In implementation, multiple physical data tables can be associated and encapsulated according to business logic to form a wide-table business object logical view. For example, the "Customer Information Table," "Order Table," and "Product Table" can be associated and encapsulated into a "Sales Details View," which belongs to the business object logical view. Subsequently, the large language model can easily generate standardized query statements based on the predefined, semantically clear atomic business objects in the business object logical view, without needing to deduce complex inter-table JOIN relationships. This reduces the complexity and error rate of query generation, improving efficiency and accuracy. The business object logical view can also be called a database logical model, belonging to the business semantic layer.

[0036] After obtaining the query elements, candidate business object logical views can be determined from several business object logical views based on the query elements. The candidate business object logical views can be business object logical views that include each query element from the several business object logical views, thereby ensuring that the recalled candidate business object logical views are highly relevant to the user's query intent.

[0037] Step S103: Based on the metadata of the logical view of the candidate business object and the data query question, generate query statement prompts.

[0038] The metadata of a candidate business object logical view can include the name of the business object logical view, the corresponding database syntax type (which can be called the database syntax type), the names and descriptions of each field, etc.

[0039] After obtaining the metadata of the logical view of the candidate business object, the metadata, data query questions, and preset system prompts can be assembled to obtain query statement generation prompts.

[0040] System prompts are structured prompts that can be used to guide large language models to generate standardized query statements (which can be SQL statements) based on the provided metadata context. System prompts can specifically emphasize that the generated standardized query statements must strictly match the database syntax type in the metadata.

[0041] For example, the query statement can generate suggestion words as follows:

[0042] "You are a multi-database SQL generation expert. Please dynamically generate SQL statements with the corresponding syntax based on the [Database Syntax Type] in the metadata, and strictly adhere to the following rules:"

[0043] I. Rule List:

[0044] 1. Output format requirements:

[0045] The generated SQL must use ### <sql>### Code block wrapping;

[0046] If the metadata cannot meet the requirements, return the message: "The information you are looking for does not exist in the database."

[0047] 2. Syntax Adaptation Principle:

[0048] Select the corresponding syntax based on the [Database Syntax Type] in the metadata header;

[0049] Table aliases are prohibited;

[0050] Time processing rules (key differences):

[0051] Timestamp Conversion:

[0052] The timestamp in database ① is converted to:

[0053] Ten-digit timestamp: FROM_UNIXTIME(timestamp_column);

[0054] 13-digit timestamp: FROM_UNIXTIME(timestamp_column / 1000);

[0055] The timestamps in database ② are converted to:

[0056] Ten-digit timestamp: To_TIMESTAMP(timestamp_column);

[0057] Thirteen-digit timestamp: T0_TIMESTAMP(timestamp_column / 1000);

[0058] The timestamp in database ③ is converted to:

[0059] Ten-digit timestamp: To_DATE('1970-01-01','YYYY-MM-DD')+(timestamp column / 86400);

[0060] 13-digit timestamp: To_DATE('1970-01-01','YYYY-MM-DD')+(timestamp column / 86400000);

[0061] 3. Field aliasing rules:

[0062] Final header generation logic:

[0063] If the field in the metadata is not null, use the field alias.

[0064] The field has an empty alias in the metadata → use the field description (in Chinese);

[0065] Calculated fields → Use Chinese descriptions of your query requirements;

[0066] Databases ② and ③ require Chinese aliases to be enclosed in double quotes: "alias";

[0067] 4. Example data description:

[0068] The example values ​​are for formatting purposes only and should not be used as search criteria.

[0069] The syntactic differences between different databases are illustrated in the examples;

[0070] Please generate SQL according to the rules above and the example. If the requirements cannot be met, please return the specified statement directly. Perform a feasibility verification first; only generate the final SQL after successful verification. The final SQL must be wrapped in a specified identifier for automated extraction. The following are the current database metadata and query requirements:

[0071] II. Database Metadata:

[0072] Table name: zhl test sales;

[0073] Database syntax type: The syntax type of database ②;

[0074] Field list:

[0075] (Field name: num; Field description: Product sales quantity; Field value type: INT; Is it a primary key: No);

[0076] (Field name:prov; Field description: Province; Field value type: VARCHAR(100); Is it a primary key: No);

[0077] (Field name: city; Field description: City name; Field value type: VARCHAR(100); Is it a primary key: Yes);

[0078] (Field name: create date; Field description: Sales time; Field value type: TIMESTAMP; Is it a primary key: No);

[0079] (Field name: product_id; Field description: Product ID; Field value type: VARCHAR(100); Is it a primary key: No);

[0080] (Field name: city_code; Field description: city code; Field value type: VARCHAR(100); Is it a primary key: No);

[0081] (Field name: mm132hail create date; Field description: Creation time; Field value type: TIMESTAMP; Is it a primary key: No);

[0082] III. Query Requirements:

[0083] The average sales volume of products with the ID "xxx" last year.

[0084] Step S104: Generate prompts from the query statement and input them into the large language model.

[0085] After obtaining the query suggestion terms, these terms can be input into the large language model. The query suggestion terms can be used to instruct the large language model to generate standardized queries that match the database syntax type carried by the metadata.

[0086] Step S105: Send the standardized query statement to the unified query layer.

[0087] The solution provided in this application also includes a unified query layer. The goal of the unified query layer can include shielding the syntax and storage differences of the underlying heterogeneous physical data sources and providing a standardized data access interface. In implementation, the unified query layer can be a syntax-compatible engine responsible for receiving standardized query statements and transparently translating and routing them to the corresponding physical data sources for execution. The unified query layer can decouple the upper-layer semantic parsing logic from the details of the underlying physical storage, improving the system's scalability (supporting the addition of new data sources) and maintainability.

[0088] After obtaining the standardized query statement output by the large language model, it can be sent to the unified query layer. The standardized query statement instructs the unified query layer to translate it, routing the translated statement to the physical data source corresponding to the database syntax type for execution. This yields the data query results corresponding to the data query question. The data query results can then be presented to the user through a front-end interface.

[0089] In the above data query processing method, the user's data query question is input into the large language model; the data query question instructs the large language model to extract query elements from the data query question; based on the query elements, candidate business object logical views are determined from several pre-built business object logical views; the business object logical views are obtained by semantically abstracting several physical data tables based on business logic; the physical data tables are the data tables stored in the physical data source; based on the metadata of the candidate business object logical views and the data query question, query statement generation prompts are obtained; the query statement generation prompts are input into the large language model; the query statement generation prompts instruct the large language model to generate a standardized query statement that matches the database syntax type carried by the metadata; the standardized query statement is sent to the unified query layer; the standardized query statement instructs the unified query layer to translate the standardized query statement, and routes the translated standardized query statement to the physical data source corresponding to the database syntax type for execution, thereby obtaining the data query result corresponding to the data query question. In the solution provided in this application, users can provide data query questions in natural language, and the corresponding data query results can be automatically and accurately returned to the user, lowering the barrier to entry for data querying. Furthermore, leveraging the natural language processing capabilities of a large language model, this application can automatically extract relatively accurate query elements from the data query questions without manual intervention. Based on business logic, semantic abstraction is performed on several physical data tables to obtain logical views of business objects. The establishment of these logical views improves the generation of multi-level nested subqueries or query statements involving complex multi-table joins (such as SQL), enhancing complex query capabilities. Moreover, based on the query elements, candidate logical views of business objects can be determined from several pre-built logical views. Based on the metadata of the candidate logical views and the data query questions, query statement generation prompts can be obtained. These prompts are then input into the large language model, and leveraging its natural language understanding capabilities, the accuracy of combined queries using aggregate functions (such as COUNT, SUM, AVG) and GROUP BY clauses can be improved, ensuring the accuracy of sorting constraints (such as ORDER). The correct generation of BY and LIMIT can enhance the ability to perform complex queries. In addition, after the large language model generates standardized query statements that match the database syntax type carried by the metadata, the unified query layer can translate them and route them to the corresponding physical data source for execution. This allows the large language model to reduce the number of join operations on physical data tables during the data query statement generation stage, thereby improving accuracy.

[0090] In one embodiment, based on the query elements, candidate business object logical views are determined from a pre-built set of business object logical views, including:

[0091] Obtain the semantic vectors and text indexes of the field meaning texts in the logical views of each business object; match the query elements with the text indexes of the field meaning texts to determine the first-round recall results in several business object logical views; match the semantic vectors of the query elements with the semantic vectors of the field meaning texts to determine the second-round recall results in several business object logical views; obtain the candidate business object logical views according to the first-round recall results and the second-round recall results.

[0092] A dedicated text embedding model can be pre-constructed. The text embedding model can use model_m3e, and model_m3e can be abbreviated as M3E. Model_m3e is an open-source pre-trained model based on the Transformer architecture. It can be fine-tuned according to specific business data, so as to train a semantic understanding model dedicated to database field names and field description texts. This semantic understanding model can more accurately convert database-related text information into high-quality semantic vectors.

[0093] The fine-tuning scheme can include dataset construction. When constructing the dataset, a scheme of business data + large model generation can be adopted, so as to construct 100,000 sample pairs of (Chinese field meaning, query elements); the label of each sample pair can be determined through manual annotation, where 1 represents semantic similarity and 0 represents semantic difference, as shown in Table 1. Exemplarily, the user's data query problem can be "query the long-term insurance premium in Area A last year", and the extracted query elements include "year", "province", and "long-term insurance premium". The Chinese field meaning can refer to the Chinese alias corresponding to the English field name in the business data. For example, the Chinese meaning of the "province" field is "province". In the actual execution process of the algorithm layer, the Chinese alias can be used to participate in the similarity matching calculation.

[0094] Table 1

[0095]

[0096] The model structure used in the fine-tuning scheme can be a two-tower model constructed based on model_m3e. The structure of the two-tower model is as Figure 2 As shown, the dual-tower model can share parameters. The Chinese meaning of the field and the question element can be processed through the embedding layer and the encoding layer respectively to obtain the corresponding semantic vectors. The cosine distance between the semantic vector of the Chinese meaning of the field and the semantic vector of the question element can be calculated, and then the cross-entropy loss can be calculated with the label. Finally, the model parameters are updated through backpropagation. The stably converged model can be used for the construction of subsequent vector indices. The embedding layer can include position embedding units, token embedding units, and segment embedding units. There can be N encoding layers, which can employ a Transformer encoder. The Transformer encoder can include Multi-Head Attention (MATN) units, Add & Normal (LFN) units, and FFN (Feedforward Neural Network) units.

[0097] For business object logical views that support natural language queries, the Chinese meaning of each field in the business object logical view can be extracted, and then a dual-path index can be built. This enables the construction of a multimodal index, providing an efficient retrieval foundation for subsequent semantic and keyword-based model recall. The dual-path index can include vector indexes, where the Chinese meaning text of the fields can be converted into 512-dimensional semantic vectors using the model_m3e model, and stored in a vector database. The dual-path index can also include text indexes, which can be built using the Chinese meaning text of the fields.

[0098] After obtaining the question elements output by the large language model, each question element can be retrieved separately:

[0099] (1) Text index recall: Full-text search is performed in the text index using the plaintext of the query element, so that the first recall result can be obtained in several business object logical views.

[0100] (2) Vector index recall: The model_m3e model can be used to convert the query elements into semantic vectors with a length of 512 dimensions, and similarity retrieval can be performed in the vector database, so that the second recall result can be obtained in several business object logical views.

[0101] The solution provided in this application also includes a fusion and filtering algorithm (merge algorithm), which can merge the first-way recall results and the second-way recall results. The filtering logic can include that the selected candidate business object logical view must contain fields that can cover each query element (that is, the candidate business object logical view must contain the fields of each query element), thereby ensuring that the recalled candidate business object logical view is highly relevant to the user's query intent.

[0102] In one embodiment, the semantic vector of the query element is matched with the semantic vector of the field meaning text to determine the second-way recall result in several business object logical views, including:

[0103] Calculate the cosine distance between the semantic vector of the query element and the semantic vector of the field meaning text; if the cosine distance meets the condition, then obtain the second-way recall result based on the business object logical view containing the field meaning text.

[0104] When using vector indexing for recall, cosine distance can be used as a similarity calculation method. Specifically, the cosine distance between the semantic vector of the query element and the semantic vector of the field meaning text can be calculated. If the cosine distance meets the condition, the second recall result is obtained based on the business object logical view containing the field meaning text.

[0105] In one embodiment, a standardized query statement is obtained by the large language model after generating an initial query statement that matches the database syntax type carried by the metadata, and then performing error correction and optimization on the initial query statement.

[0106] To ensure the accuracy and robustness of standardized query statements output by the large language model, query statement correction and optimization units can be used. In implementation, the capabilities of the large language model can be leveraged again to perform error correction and optimization on the initial query statements generated and matched with the database syntax types carried by the metadata. This results in standardized query statements with higher accuracy and robustness, improving the final query success rate and efficiency.

[0107] The optimization process can include performance optimization, which can be used to analyze, suggest, or directly apply possible query optimization strategies (such as adding index hints, rewriting subqueries, etc.).

[0108] In one embodiment, error correction processing includes at least one of syntax error correction processing and semantic error correction processing.

[0109] Syntax correction and semantic correction can be used to check for and correct potential logical or syntactic errors.

[0110] In one embodiment, translating a standardized query statement includes:

[0111] Based on the relationships between the physical data tables that form the logical view of the business object, the standardized query statement is translated; the standardized query statement corresponds to the logical view of the business object, and the translated standardized query statement corresponds to the physical data table.

[0112] The logical view of a business object can be understood as a large wide table, each of which can be composed of several related physical data tables. The standardized query statements output by the large language model are based on the large wide table. After the standardized query statements are passed to the unified query layer, the relationships between the several related physical data tables that make up the large wide table can be used to translate the standardized query statements. The translated standardized query statements correspond to the physical data tables and can be executed directly on the physical data tables. This allows the large language model to reduce the number of join operations on physical data tables during the data query statement generation stage, thereby improving accuracy.

[0113] To better understand the above method, an application example of the data query processing method of this application is described in detail below. The query statement in this application example can be SQL (Structured Query Language). In related technologies, the database usage process has the following problems:

[0114] (1) Challenges of Manual SQL Writing. Data queries heavily rely on business personnel manually writing SQL. Complex queries have a high technical threshold, resulting in low analysis efficiency. Non-technical personnel find it difficult to master quickly, which seriously affects the speed of data acquisition.

[0115] (2) Limitations of intelligent tools. Intelligent tools have long response cycles and cannot meet the needs for immediacy and conversational questioning. In a fast-paced business environment, the delay in data acquisition leads to delayed decision-making.

[0116] (3) Data silo problem. Data can be stored through multiple heterogeneous physical data sources. Cross-database queries are inefficient, and the data silo phenomenon is serious, which restricts the rapid release of data value. There is no data query solution that is zero-threshold, highly efficient, and unified across sources.

[0117] This application example provides an intelligent query solution based on large model and multi-source data fusion. It enables users to ask questions in natural language, and automatically, accurately, and efficiently generate and execute optimized SQL queries across heterogeneous data sources, returning the required data query results. This reduces the barrier to entry, improves query efficiency and accuracy, and enhances system flexibility and scalability.

[0118] This application example demonstrates how to build a layered, integrated, and intelligently driven natural language query system that crosses heterogeneous data sources. This system can be structured around several aspects, including unified data access, semantic abstraction, intelligent model matching, and accurate SQL generation, and mainly comprises the following units:

[0119] (1) Unified query layer (also known as unified query and data fusion layer):

[0120] Objective: To shield the differences in syntax and storage between underlying heterogeneous physical data sources and provide a standardized data access interface.

[0121] Implementation: A unified query layer can be built based on the database syntax type of a certain physical data source. The core of the unified query layer can be a syntax-compatible engine responsible for receiving standardized SQL and transparently translating and routing it to the corresponding physical data source for execution. The unified query layer can decouple the upper-layer semantic parsing logic from the details of the underlying physical storage, thereby improving the system's scalability (supporting the addition of new data sources) and maintainability.

[0122] (2) Logical semantic abstraction layer:

[0123] Objective: To abstract the complex, underlying physical data table structure and its relationships into a business-friendly logical view (which can be called a business object logical view), simplifying the subsequent SQL generation process.

[0124] Implementation: Construct a logical view of business objects (business semantic layer). By associating and encapsulating multiple physical tables according to business logic, a wide-table logical view of business objects is formed. For example, the "Customer Information Table," "Order Table," and "Product Table" are associated and encapsulated into a "Sales Details View," which belongs to the logical view of business objects. The subsequent intelligent algorithm layer can easily generate standardized query statements based on the predefined, semantically clear atomic business objects in the logical view of business objects, without the need to deduce complex inter-table JOIN relationships. This reduces the complexity and error rate of query statement generation, and improves generation efficiency and accuracy.

[0125] (3) Intelligent Algorithm Layer:

[0126] The intelligent algorithm layer is one of the core components of system intelligence, enabling the transformation of user natural language data query questions into standardized SQL.

[0127] a. Dedicated text embedding model construction (model_m3e):

[0128] Based on the open-source M3E, fine-tuning model_m3e on specific business data allows for the training of a semantic understanding model specifically for database field names and descriptions. This semantic understanding model can more accurately convert database-related text information into high-quality semantic vectors.

[0129] The fine-tuning solution may include dataset construction. When constructing the dataset, a solution combining business data and large model generation can be adopted, so that 100,000 sample pairs (Chinese meanings of fields, question elements) can be constructed; through manual annotation, the label of each sample pair can be determined, where 1 represents semantic similarity and 0 represents semantic difference. Exemplarily, the user's data query question can be "Query the long-term insurance premium in Area A last year", and the extracted question elements include "year", "province", and "long-term insurance premium". The Chinese meaning of the field can refer to the Chinese alias corresponding to the English field name in the business data. For example, the Chinese meaning of the "province" field is "省份". In the actual execution process of the algorithm layer, the Chinese alias can be used to participate in the similarity matching calculation.

[0130] The model structure used in the fine-tuning solution can be a two-tower model based on model_m3e. The structure of the two-tower model is as Figure 2 shown. The two-tower model can share parameters. The Chinese meaning of the field and the question element can respectively pass through the embedding layer and the encoding layer. After obtaining the corresponding semantic vectors, the cosine distance between the semantic vector of the Chinese meaning of the field and the semantic vector of the question element can be calculated, and then the cross-entropy loss can be calculated with the label. Furthermore, the model parameters can be updated through backpropagation. The model with stable convergence can be used for the subsequent construction of vector indexes.

[0131] b. Multimodal index construction:

[0132] For the business object logical view that plans to support natural language queries, the Chinese meanings of each field in the business object logical view can be extracted, and then a two-way index can be constructed, so that the construction of the multimodal index can be realized, which can provide an efficient retrieval basis for subsequent model recall based on semantics and keywords.

[0133] The two-way index can include a vector index. The model_m3e model can be used to convert the Chinese meaning text of the field into a 512-dimensional semantic vector, which can be stored in the vector database. The two-way index can include a text index, and a text index can be constructed with the Chinese meaning text of the field.

[0134] c. User intent parsing and main word extraction:

[0135] Using a large language model, analyze the natural language data query question input by the user. The task of the large language model can be to identify and extract the question elements (main words) in the data query question, and can also classify the entity types of the question elements. For example, the entity type of "Area A" can be a city, the entity type of "last year" can be a year, and the entity type of "total population" can be a population indicator. Through the large language model, the core semantics of the user's query can be captured more accurately.

[0136] d. Intelligent recall of logical views of candidate business objects:

[0137] Each extracted question element can be retrieved separately:

[0138] 1) Text Index Recall: Full-text search is performed on the text index using the plaintext of the query elements, thus obtaining the first-way recall results in several logical views of business objects. The text index can be stored in an Elasticsearch (ES) database, which is an open-source database.

[0139] 2) Vector Index Recall: The model_m3e model can be used to convert query elements into 512-dimensional semantic vectors. Similarity retrieval is then performed in a vector database (using cosine distance as the similarity calculation method), thus obtaining a second-path recall result from several business object logical views. The vector database can be, for example, but not limited to, the Milvus vector database, which is an open-source vector database.

[0140] Merge Algorithm: This algorithm can merge the first-way recall results and the second-way recall results. The filtering logic can include ensuring that the selected candidate business object logical view contains fields that can cover each query element (that is, the candidate business object logical view must contain the fields of each query element), thereby ensuring that the recalled candidate business object logical view is highly relevant to the user's query intent.

[0141] e. Context-aware Text2SQL generation:

[0142] The metadata of the candidate business object logical view obtained in the previous step, the data query question, and the system suggestion words are assembled to obtain query statement generation suggestions. These suggestions can be input into a large language model to generate standardized SQL.

[0143] f. SQL optimization and error correction:

[0144] To ensure the accuracy and robustness of SQL, an SQL error correction and optimization unit can be used. The capabilities of a large language model can be leveraged again to perform syntax / semantic error correction and performance optimization on the generated initial SQL, thereby improving the final query success rate and efficiency.

[0145] Syntax / semantic correction can be used to check for and correct potential logical or syntactic errors. Performance optimization can be used to analyze and suggest or directly apply possible query optimization strategies (such as adding index hints, rewriting subqueries, etc.).

[0146] (4) Query execution and result delivery:

[0147] Optimized and validated standardized SQL can be adapted to the corresponding physical data source for execution through a unified query layer syntax compatibility engine. The system prompts explicitly specify the database syntax type; therefore, the standardized SQL generated by the large language model is compatible with the syntax of the corresponding physical data source, ensuring smooth execution. Data query results can be returned to the front-end interface and presented to the user.

[0148] like Figure 3 As shown, the workflow of this application example may include:

[0149] User inputs a data query question → Large Language Model extracts and categorizes query elements → Dual-path indexes retrieve relevant fields in parallel → Candidate business object logical views covering each query element are selected → Candidate business object logical views, data query questions, and system prompts are assembled → Large Language Model generates initial SQL → SQL error correction and optimization units verify and correct the SQL → The optimized standardized SQL is executed through a unified query layer → Data query results are returned to the user. The Large Language Model (LLM) is the full name of this query.

[0150] This application example enhances the processing capabilities of complex queries and alleviates semantic understanding and ambiguity resolution issues. Specifically, in terms of improving complex query processing capabilities, the construction of logical views of business objects improves the generation of SQL queries involving multi-level nested subqueries or complex multi-table joins; it improves the performance of combined queries using aggregate functions such as COUNT / SUM / AVG and GROUP BY clauses; and it improves the correct generation of sorting constraints such as ORDER BY and LIMIT. When the database schema (table structure) is complex, it can more accurately understand the relationships between tables, better adapting to complex cross-domain scenarios; and it can also achieve automatic syntax adaptation for heterogeneous physical data sources. Regarding alleviating semantic understanding and ambiguity resolution issues, the use of query element extraction and a dual-path recall mechanism solves the difficulty in recognizing inconsistencies between user natural language terms and database field names; in terms of quantifier and comparison processing, it improves the accuracy of SQL conversion for vague quantifiers such as "at least" and "no more than"; and in terms of time expressions, it reduces the error rate in converting relative time expressions such as "last month" and "last year".

[0151] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0152] Based on the same inventive concept, this application also provides a data query processing apparatus for implementing the data query processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data query processing apparatus embodiments provided below can be found in the limitations of the data query processing method described above, and will not be repeated here.

[0153] In one embodiment, such as Figure 4 As shown, a data query processing device is provided, comprising:

[0154] The question input module 401 is used to input the user's data query question into the large language model; the data query question is used to instruct the large language model to extract the query elements from the data query question;

[0155] The view determination module 402 is used to determine candidate business object logical views from a number of pre-built business object logical views based on the query elements; the business object logical views are obtained by semantically abstracting a number of physical data tables based on business logic; the physical data tables are data tables stored in physical data sources.

[0156] The prompt word determination module 403 is used to generate prompt words based on the metadata of the logical view of the candidate business object and the data query question;

[0157] The prompt word input module 404 is used to input the prompt word generated by the query statement into the large language model; the prompt word generated by the query statement is used to instruct the large language model to generate a standardized query statement that matches the database syntax type carried by the metadata;

[0158] The statement sending module 405 is used to send the standardized query statement to the unified query layer; the standardized query statement is used to instruct the unified query layer to translate the standardized query statement, route the translated standardized query statement to the physical data source corresponding to the database syntax type for execution, and obtain the data query result corresponding to the data query question.

[0159] In one embodiment, the view determination module 402 is configured to:

[0160] Obtain the semantic vector and text index of the field meaning text in each of the business object logical views; match the question element with the text index of the field meaning text to determine a first recall result in several business object logical views; match the semantic vector of the question element with the semantic vector of the field meaning text to determine a second recall result in several business object logical views; obtain candidate business object logical views based on the first recall result and the second recall result.

[0161] In one embodiment, the view determination module 402 is configured to:

[0162] Calculate the cosine distance between the semantic vector of the question element and the semantic vector of the field meaning text; if the cosine distance satisfies the condition, then obtain the second recall result based on the business object logical view containing the field meaning text.

[0163] In one embodiment, the standardized query statement is obtained by the large language model after generating an initial query statement that matches the database syntax type carried by the metadata, and then performing error correction and optimization processing on the initial query statement.

[0164] In one embodiment, the error correction process includes at least one of syntax error correction and semantic error correction.

[0165] In one embodiment, the unified query layer is used for:

[0166] The standardized query statement is translated based on the relationship between the physical data tables that form the logical view of the business object; the standardized query statement corresponds to the logical view of the business object, and the translated standardized query statement corresponds to the physical data table.

[0167] Each module in the aforementioned data query and processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0168] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores the data involved in the aforementioned methods. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data query processing method.

[0169] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the various method embodiments described above.

[0171] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0172] In one embodiment, a computer program product is provided having a computer program stored thereon, the computer program being executed by a processor of the steps described in the various method embodiments above.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0174] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0176] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / sql>

Claims

1. A data query processing method, characterized in that, The method comprises: inputting a data query question of a user into a large language model; the data query question is used to instruct the large language model to extract a question element from the data query question; determining a candidate business object logical view from a plurality of pre-constructed business object logical views according to the question element; the business object logical view is obtained by performing semantic abstraction on a plurality of physical data tables according to business logic; the physical data table is a data table stored in a physical data source; obtaining a query statement generation prompt word according to metadata of the candidate business object logical view and the data query question; inputting the query statement generation prompt word into the large language model; the query statement generation prompt word is used to instruct the large language model to generate a standardized query statement matching a database syntax type carried by the metadata; sending the standardized query statement to a unified query layer; the standardized query statement is used to instruct the unified query layer to translate the standardized query statement, and route the translated standardized query statement to a physical data source corresponding to the database syntax type to execute, so as to obtain a data query result corresponding to the data query question.

2. The method of claim 1, wherein, Determining a candidate business object logical view from a plurality of pre-constructed business object logical views according to the question element comprises: obtaining semantic vectors and text indexes of field meaning texts in each business object logical view; matching the question element with the text indexes of the field meaning texts to determine a first route recall result in the plurality of business object logical views; matching semantic vectors of the question element with semantic vectors of the field meaning texts to determine a second route recall result in the plurality of business object logical views; obtaining the candidate business object logical view according to the first route recall result and the second route recall result.

3. The method of claim 2, wherein, Matching semantic vectors of the question element with semantic vectors of the field meaning texts to determine a second route recall result in the plurality of business object logical views comprises: calculating a cosine distance between the semantic vectors of the question element and the semantic vectors of the field meaning texts; if the cosine distance meets a condition, obtaining the second route recall result according to a business object logical view containing the field meaning text.

4. The method of claim 1, wherein, The standardized query statement is obtained by performing error correction processing and optimization processing on an initial query statement generated by the large language model to match the database syntax type carried by the metadata.

5. The method of claim 4, wherein, The error correction processing comprises at least one of syntax error correction processing and semantic error correction processing.

6. The method according to any one of claims 1 to 5, characterized in that, Translating the standardized query statement comprises: translating the standardized query statement according to relationships between physical data tables forming the business object logical view; the standardized query statement corresponds to the business object logical view, and the translated standardized query statement corresponds to the physical data table.

7. A data query processing apparatus, characterized by comprising: The device comprises: The question input module is configured to input a data query question of a user into the large language model, and the data query question is configured to instruct the large language model to extract a question element from the data query question. The view determination module is configured to determine a candidate business object logical view from a plurality of business object logical views that are pre-constructed according to the question element, wherein the business object logical view is obtained by performing semantic abstraction on a plurality of physical data tables according to business logic, and the physical data table is a data table stored in a physical data source. The prompt word determination module is configured to obtain a query statement generation prompt word according to metadata of the candidate business object logical view and the data query question. The prompt word input module is configured to input the query statement generation prompt word into the large language model, and the query statement generation prompt word is configured to instruct the large language model to generate a standardized query statement that matches a database syntax type carried by the metadata. The statement sending module is configured to send the standardized query statement to a unified query layer, and the standardized query statement is configured to instruct the unified query layer to translate the standardized query statement, route the translated standardized query statement to a physical data source corresponding to the database syntax type, and execute the translated standardized query statement to obtain a data query result corresponding to the data query question.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images