Database operation statement generation method and device, electronic equipment and medium

Abstract

The invention provides a database operation statement generation method and device, electronic equipment and a medium. The method comprises the following steps: based on multi-dimensional information of a target table, performing similarity retrieval in a vector database to obtain a candidate table set; according to the table structure and the business semantics of each candidate table in the candidate table set and the table structure and the business semantics of the target table, determining a first cue word used for screening the candidate tables according to the table structure and the business semantics; screening out a target candidate table from the candidate table set according to the first cue word by utilizing a large language model; according to the data definition language and the business semantics of the target table and the table structure and the business semantics of the target candidate table, a second cue word used for generating a database operation statement is determined, and the database operation statement is used for inserting data in the target candidate table into the target table; and generating a database operation statement according to the second cue word by using the large language model.

Description

Technical Field

[0001] The present specification relates to the field of big data technology, and in particular to a method, device, electronic device and medium for generating database operation statements. Background Art

[0002] With the rapid expansion of enterprise data and the increasing complexity of business scenarios, traditional data governance technologies are facing increasing challenges, especially in the process of data table selection and SQL generation. These processes rely on users' deep understanding of database structure and business rules, and are difficult to cope with the complexity and dynamic changes of multi-dimensional information, resulting in low efficiency and high operational difficulty. Summary of the invention

[0003] In order to overcome the problems existing in the related art, this specification provides a method, device, electronic device and medium for generating a database operation statement.

[0004] According to a first aspect of an embodiment of the present specification, a method for generating a database operation statement is provided, the method comprising: performing similarity search in a vector database based on multidimensional information of a target table to obtain a set of candidate tables; determining a first prompt word for screening candidate tables according to the table structure and business semantics of each candidate table in the set of candidate tables, and the table structure and business semantics of the target table; using a large language model, screening a target candidate table from the set of candidate tables according to the first prompt word; determining a second prompt word for generating a database operation statement according to a data definition language and business semantics of the target table, and the table structure and business semantics of the target candidate table, the database operation statement being used to insert data from the target candidate table into the target table; and using the large language model, generating a database operation statement according to the second prompt word.

[0005] According to a second aspect of an embodiment of the present specification, a device for generating a database operation statement is provided, comprising: a retrieval module, configured to perform similarity retrieval in a vector database based on multidimensional information of a target table to obtain a set of candidate tables; a first prompt word determination module, configured to determine a first prompt word for screening candidate tables according to the table structure and business semantics of each candidate table in the set of candidate tables, and the table structure and business semantics of the target table; a screening module, configured to use a large language model to screen a target candidate table from the set of candidate tables according to the first prompt word; a second prompt word determination module, configured to determine a second prompt word for generating a database operation statement according to the data definition language and business semantics of the target table, and the table structure and business semantics of the target candidate table, the database operation statement being used to insert data from the target candidate table into the target table; and a generation module, configured to use a large language model to generate a database operation statement according to the second prompt word.

[0006] According to a third aspect of the embodiments of this specification, there is provided an electronic device, including:

[0007] processor;

[0008] a memory for storing processor-executable instructions;

[0009] The processor is configured to execute the method for generating a database operation statement according to the first aspect or any corresponding implementation manner thereof.

[0010] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, including: computer instructions are stored on the computer-readable storage medium, and the computer instructions are used to enable a computer to execute the method for generating database operation statements of the above-mentioned first aspect or any corresponding implementation method thereof.

[0011] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0012] In the embodiments of this specification, natural language processing technology is combined with multi-dimensional vector retrieval technology to automatically convert the user's natural language queries into database operation statements and intelligently select appropriate data tables, thereby greatly reducing the user's understanding requirements for databases and business logic, allowing non-technical users to efficiently perform complex data queries.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0015] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0016] Figure 2 The flowchart of a method for generating a database operation statement is shown in this specification according to an exemplary embodiment.

[0017] Figure 3 It is a hardware structure diagram of the computer device where the device for generating database operation statements in the embodiment of this specification is located.

[0018] Figure 4 It is a block diagram of a device for generating a database operation statement according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0020] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0022] Next, the embodiments of this specification are described in detail.

[0023] The following combination Figure 1The system architecture of the method and device for generating database operation statements applicable to the embodiments of this specification is described. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of this specification can be applied, to help those skilled in the art understand the technical content of this specification, but it does not mean that the embodiments of this specification cannot be used in other devices, systems, environments or scenarios.

[0024] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0025] like Figure 1 As shown, the system architecture may include, for example, a terminal device, a network and a server. The network is used to provide a medium for a communication link between the terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0026] Users can use terminal devices to interact with servers through the network to receive or send messages, etc. Various communication client applications can be installed on the terminal devices, such as artificial intelligence software, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc.

[0027] The terminal device may be any electronic device having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0028] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the terminal device. The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as the web page, information, or data obtained or generated according to the user request) to the terminal device.

[0029] The following is a detailed description of the method for generating a database operation statement provided in the embodiment of this specification. Figure 2 As shown, Figure 2 This is a flowchart of a method for generating a database operation statement according to an exemplary embodiment of this specification. The method can be applied to systems such as servers. The method for generating a database operation statement provided in the embodiment of this specification may include the following steps.

[0030] In step 210, based on the multi-dimensional information of the target table, a similarity search is performed in the vector database to obtain a set of candidate tables.

[0031] According to an embodiment of the present specification, the multi-dimensional information may include information of multiple dimensions of the target table, such as table name, description information, table structure, and additional business information. The vector database may include table vectors corresponding to multiple data tables. The table vector may be obtained, for example, by vectorizing the multi-dimensional information of the data table.

[0032] In step 220, based on the table structure and business semantics of each candidate table in the candidate table set and the table structure and business semantics of the target table, a first prompt word for screening the candidate tables based on the table structure and business semantics is determined.

[0033] According to an embodiment of the present specification, the first prompt word may be used to instruct the large model to select a candidate table that best matches the target table from the candidate table set based on dimensions such as table structure and business semantics. The large model may include, for example, a large language model (LLM).

[0034] In step 230, a target candidate list is screened from the candidate list set according to the first prompt word using the large language model.

[0035] According to an embodiment of the present specification, for example, the first prompt word may be input into the large model. The large model may analyze and process the candidate tables in the candidate table set according to the first prompt word, and select a target candidate table.

[0036] According to the embodiments of this specification, based on the multi-dimensional information of the target table, similarity search is performed in the vector library to preliminarily screen out a group of candidate tables. Then, the table structure and business semantics of these candidate tables are comprehensively evaluated through the big model to screen out the candidate table that best meets the requirements of the target table. In this way, the structural similarity and business scenario matching are combined to ensure the dual fit of the data table at the technical structure and business level.

[0037] In step 240, a second prompt word for generating a database operation statement is determined based on the data definition language and business semantics of the target table and the table structure and business semantics of the target candidate table. The database operation statement is used to insert the data in the target candidate table into the target table.

[0038] According to an embodiment of the present specification, the second prompt word can be used to instruct the large model to generate a database operation statement based on the table structure and business semantics of the target table and the target candidate table, and the database operation statement can be used to insert the data in the target candidate table into the target table. Exemplarily, in this embodiment, the database operation statement can include an SQL statement.

[0039] In step 250, a database operation statement is generated according to the second prompt word using the large language model.

[0040] According to an embodiment of the present specification, for example, the second prompt word can be input into the large model. The large model can perform natural language processing based on the second prompt word to generate a corresponding database operation statement. Natural language processing can include text2sql (text to SQL) processing, for example.

[0041] Related technologies In the process of generating database operation statements, users need to have a deep understanding of database structure and business logic, which has a high technical threshold for users who lack professional background.

[0042] According to the embodiments of the present specification, through multi-dimensional vector retrieval technology, the user's natural language query can be automatically converted into database operation statements, and the appropriate data table can be intelligently selected, thereby greatly reducing the user's understanding requirements for the database and business logic, allowing non-technical users to perform complex data queries efficiently.

[0043] Optionally, the multidimensional information may include a table name, description information, table structure, and additional business information. Among them, the table name may be a unique identifier of a data table. The description information may include a description text of the data table. The table structure may include DDL (Data Definition Language) information of the data table. The additional business information may include tags, themes, layers, etc., which reflect the application of the data table in a specific business scenario and its background. In this embodiment, the multidimensional information may be a natural language that conforms to a predetermined format. For example, the table name, description information, table structure, and additional business information of the target table may be textualized to be converted into a natural language that conforms to a predetermined format to obtain the multidimensional information of the target table.

[0044] Based on this, step 210 may include, for example: vectorizing the table name and description information to obtain a first search vector; calculating the first similarity between the first search vector and each table vector in the vector database; determining a first table vector in the vector database whose first similarity is greater than a first threshold, and the first table vector corresponds to the first data table; wherein the first threshold can be set according to actual needs. Vectorizing the table structure to obtain a second search vector; calculating the second similarity between the second search vector and each table vector in the vector database; determining a second table vector in the vector database whose second similarity is greater than the first threshold, and the second table vector corresponds to the second data table; vectorizing the additional business information to obtain a third search vector; calculating the third similarity between the third search vector and each table vector in the vector database; determining a third table vector in the vector database whose third similarity is greater than the first threshold, and the third table vector corresponds to the third data table; merging the first data table, the second data table, and the third data table to obtain a candidate table set.

[0045] Optionally, a corresponding weight can be set for each dimension in advance. Based on this, for each candidate table in the candidate table set, the first similarity, the second similarity and the third similarity of the candidate table can be weighted and summed to obtain a comprehensive score of the data table. Then, the candidate tables in the candidate table set with a comprehensive score lower than the score threshold are deleted. The score threshold and weight can be set according to actual needs. By screening out candidate tables with low comprehensive scores, the quality of the candidate tables can be improved, thereby improving the accuracy and efficiency of subsequent candidate table matching.

[0046] Exemplarily, in this embodiment, the weight corresponding to the first similarity can be the first weight, the weight corresponding to the second similarity can be the second weight, and the weight corresponding to the third similarity can be the third weight. The comprehensive score of the data table can be calculated according to the following formula: Comprehensive score = (first similarity × first weight) + (second similarity × second weight) + (third similarity × third weight).

[0047] Optionally, the first prompt word may include target table information, candidate table information and task information, and the task information may be used to instruct the large model to analyze the candidate table information based on the table structure, business semantics and numerical relationship to screen out the target candidate table with the highest matching degree with the target table information. Based on, step 220 may include, for example: embedding the table structure and business semantics of the target table into the target table information in the first prompt word; embedding the table structure and business semantics of each candidate table into the candidate table information in the first prompt word.

[0048] Optionally, the second prompt word may include the data definition language and business semantics of the target table, the table structure and business semantics of the target candidate table, task information, technical requirement information and business association information. Among them, the task information can be used to instruct the big model to generate a database operation statement based on the target table information and the target candidate table information. The technical requirement information can be used to indicate the technical requirements that the big model should meet when generating the database operation statement. The technical requirements may include, for example, field matching requirements, primary and foreign key constraint requirements and data conversion logic requirements. The business association information can be used to indicate the business semantic requirements that the big model should meet when generating the database operation statement.

[0049] Optionally, in the process of determining the target candidate table, it may be encountered that the candidate table output by the large model does not fully meet the user's needs. In order to ensure that the final output meets the user's requirements, the user can provide feedback and supplementary information on the output results of the large model to gradually optimize the process of determining the target candidate table. Based on this, feedback information for the target candidate table can also be received. The feedback information may include supplementary business and technical requirements, etc. Based on the feedback information, a correction prompt word is generated. The correction prompt word can be used to guide the large model to gradually optimize the selected candidate table according to the business and technical requirements. Then, the large language model can be used to filter out a new target candidate table from the candidate table set based on the correction prompt word.

[0050] By receiving user feedback on the output results of the big model, the big model can dynamically adjust the recommendation criteria of the candidate list. The feedback and adjustment process can be repeated to gradually optimize until the final output results of the big model meet the target requirements. As a result, the flexibility and intelligence level of the candidate list recommendation are significantly improved, manual intervention is reduced, and the accuracy of the recommendation is improved.

[0051] Optionally, after generating the database operation statement, the database operation statement may be verified. By verifying, it can be ensured that the database operation statement can be correctly executed in the actual database environment. Exemplarily, in this embodiment, the verification may include at least one of the following operations:

[0052] (1) Check whether the data inserted into the target table matches the original fields of the target table when the database operation statement is executed. If the data inserted into the target table matches the original fields of the target table, it is determined that the verification has passed, otherwise it is determined that the verification has failed.

[0053] (2) Verify whether the data conversion logic in the database operation statement is correct. If the data conversion logic is correct, it is determined that the verification has passed, otherwise it is determined that the verification has failed.

[0054] (3) Verify whether the data inserted into the target table destroys the integrity and primary and foreign key constraints of the target table when the database operation statement is executed. If the integrity and primary and foreign key constraints of the target table are not destroyed, the verification is determined to be passed, otherwise the verification is determined to be failed.

[0055] Optionally, if the generated database operation statement fails to pass the verification, a multi-round dialogue mode can be entered. That is, the system can prompt the user to supplement the missing business information or technical details, such as field mapping rules, data type conversion requirements, etc., by interacting with the user. The large model generates a new database operation statement based on the new input and performs automatic verification again. The above process will be repeated until the system generates a database operation statement that meets the requirements. As a result, the automation, flexibility and accuracy of the generation of database operation statements are improved.

[0056] The following is another method for generating a database operation statement according to another exemplary embodiment, comprising the following steps:

[0057] According to the embodiments of this specification, the data table with multi-dimensional information fusion can be vectorized in advance, aiming to comprehensively and accurately collect and process the multi-dimensional information of the data table, and convert it into a structured semantic expression, and finally generate a high-dimensional vector representation that can be retrieved and matched. The process can include the following operations:

[0058] Operation 11: Data table multi-dimensional information collection. According to the embodiments of this specification, multi-dimensional information such as structure information, business information, data lineage, management information, data quality, and domain knowledge of the data table can be fully collected to ensure a comprehensive understanding of the data table.

[0059] Operation 12: Data table semantic analysis and textualization. According to the embodiments of this specification, information of different dimensions can be reasonably layered and organized in a natural language through semantic analysis and textualization, ensuring that the information can accurately reflect the structure and semantics of the data table.

[0060] Operation 13: Multi-dimensional vector information storage. According to the embodiments of this specification, the textualized data may be vectorized using a pre-trained model and stored in a vector database to provide basic support for subsequent fast matching and retrieval.

[0061] During the entire data table vectorization process, it is necessary to collect multi-dimensional information of the data table. The purpose of collecting multi-dimensional information of the data table is to comprehensively collect and integrate various types of information of each data table. This information not only provides a detailed description of the technical structure and business background of the table, but also lays the foundation for subsequent vectorization and similarity retrieval. The collected multi-dimensional information can include the following information.

[0062] (1) Table structure information. According to the embodiments of this specification, table structure information is the core part of data table information collection, which directly reflects the technical details of the data table. Through in-depth understanding of the table structure, accurate guidance can be provided for the subsequent generation of database operation statements. Table structure information may include the following elements:

[0063] Table name: A unique identifier for each table that can be used to identify the location of the data table in the database.

[0064] Field information: A detailed description of all fields in the data table, including field name, field type (such as integer, string, date, etc.), field length, etc.

[0065] Primary key and foreign key: The primary key in a table is used to uniquely identify each row of records, and the foreign key is used to associate between tables. The primary and foreign key relationship is crucial for subsequent data table association and candidate table selection.

[0066] Index: Index is an important means to improve query performance. Index information includes index type, index field, etc., which can speed up data retrieval and improve the execution efficiency of database operation statements.

[0067] Collection method: For relational databases that provide system views or information schemas, you can obtain table structure information through database operation statements, or you can use database management tools (such as DBeaver, Navicat, MySQL Workbench, etc.) to intuitively view and export table structure information.

[0068] (2)Business information.

[0069] According to the embodiments of this specification, the purpose of collecting business information is to understand the role and application scenario of the data table from the business level. This part of information can help the system to accurately select candidate tables from the business dimension in addition to the similarity of the table structure. Business information can include the following elements:

[0070] Label: The business label to which the table belongs, such as "sales data", "financial report", etc., which helps to classify the table into a specific business field.

[0071] Subject: The business subject involved in the data table, clarifying the specific role of the table in the business process.

[0072] Layering: In the data warehouse system, the layered information of the table (such as ODS layer, DWD layer, DWS layer, ADS layer, etc.) helps to understand the position and role of the table in the data processing pipeline.

[0073] Collection method: Many modern data management tools and data warehouse platforms provide corresponding functions so that users can enter and manage business information related to data tables. Through these tools, table labels, topics, and hierarchical information can be collected.

[0074] (3) Management information. According to the embodiments of this specification, management information provides management dimension information of the data table to facilitate subsequent maintenance and authority management. Clear management information can help the system make better choices in a complex database environment. Management information may include the following elements:

[0075] Creator and Responsible: The creator of the table and the team or individual responsible for maintaining it.

[0076] Department: The business department to which the data table belongs, which helps the system understand the management level and business affiliation of the data table.

[0077] Collection method: Enterprises usually use metadata management platforms to record and maintain management information of data tables. These tools provide centralized storage and query functions for information such as the creator, responsible person, and department of the data table. The metadata management platform can automatically collect information such as the creation time and maintenance personnel of the data table to facilitate management and maintenance.

[0078] (4) Data quality information.

[0079] According to the embodiments of this specification, data quality is an important factor in determining the availability of a data table. This information can help the system to perform more accurate screening when selecting candidate tables, ensuring that high-quality data is used to generate database operation statements.

[0080] Data quality information may include the following elements:

[0081] Data distribution: The distribution of field values. For example, some fields may have more missing values ​​or outliers. This information helps determine the integrity of the data.

[0082] Missing rate and outliers: For example, the proportion of missing data or outliers in a field can be counted to measure the quality of the data.

[0083] Collection method: For example, you can use specialized data quality management tools to automatically evaluate and report the quality of data tables. These tools can perform automated data analysis and generate reports on quality information such as data distribution, missing rate, and outliers. For environments where specialized data quality tools are not introduced, data quality information can be obtained through database operation statement queries and statistical functions. In a big data environment, data quality analysis can be performed by writing custom scripts.

[0084] (5) Data lineage information.

[0085] According to the embodiments of this specification, data lineage reflects the interdependence between data tables and how data flows from the source table to the target table. This information is crucial in the subsequent selection of candidate tables and helps the system determine whether the data table has data association with the target table. Data lineage information may include the following elements:

[0086] Upstream and downstream relationships: Describe the dependency between a table and other tables, such as an upstream table providing data to a table and a downstream table getting data from that table.

[0087] Cross-table dependencies: List the cross-table dependencies between a table and other systems or tables to better integrate data between tables.

[0088] Collection method: The lineage relationship of the data table is obtained through special data lineage analysis tools. These tools can automatically identify and record the flow of data.

[0089] (6)Domain knowledge base and related documents.

[0090] According to the embodiments of this specification, the domain knowledge base and related documents are supplementary information of the data table, which helps the system understand the background knowledge, business scenarios, etc. of the table. Data lineage information may include the following elements:

[0091] Related documents: including the business description, technical manual or other business and technical documents related to the data table. These documents provide background support for the generation of subsequent database operation statements and also help to understand the business background when selecting candidate tables.

[0092] Collection method: The data governance system can be equipped with a metadata management platform to centrally manage data assets and related documents. These platforms can automatically associate relevant business descriptions and technical documents through the metadata of the table (such as table name, field information, etc.).

[0093] According to the embodiments of this specification, by collecting table structure, business information, management information, data quality, data lineage and domain knowledge base, the multi-dimensional information of the data table is fully covered. These multi-dimensional information provide a complete and accurate basis for the subsequent semantic analysis and textualization of the data table and the storage of multi-dimensional vector information, and lay a solid foundation for realizing intelligent data table selection and the generation of database operation statements.

[0094] Next, the data table semantic parsing and textualization can be performed. The core task of data table semantic parsing and textualization is to convert the collected multi-dimensional information into textual descriptions, and express the specific meaning of each information dimension in a natural language. The core goal of this step is to ensure that different types of information can be expressed in a unified manner while maintaining the semantic integrity of the information, thereby improving the accuracy and flexibility of subsequent vector retrieval. Data table semantic parsing and textualization can include the following operations, for example.

[0095] Operation 21: Semantic analysis and textualization of table structure information and data lineage.

[0096] According to the embodiments of this specification, the system can perform semantic analysis on the structural information of the data table (including table name, field name, primary and foreign key relationships, indexes, etc.) and its data lineage information (such as inter-table dependencies, data sources and destinations, etc.), and textualize them in the form of natural language description. This part of information is the main reflection of the technical attributes and relationships of the table. The system stores it in the vector database by merging it into a data vector. This processing method ensures that in subsequent technical matching, the retrieval algorithm can query with the table structure and its lineage as the core, and accurately identify the data table most relevant to the user's query needs.

[0097] The semantic analysis and textualization of table structure information and data lineage may include the following steps:

[0098] Table name and field information parsing: traverse the metadata of the table and extract the table name and field name. The name of each field and its data type are converted into natural language descriptions through predefined templates. For example: original information: the table name is "Customer", the field name is "CustomerID" (integer type). Text output: "Table 'Customer' contains field 'CustomerID', which is of integer type and is the primary key of the table."

[0099] Primary and foreign key relationship analysis: Generate natural language description based on the information of primary and foreign key relationships in metadata. For example: Original information: "CustomerID" is the primary key of the main table "Customer" and is used as a foreign key in the "Order" table. Text output: "Table 'Customer' is associated with table 'Order' through 'CustomerID'."

[0100] Data lineage analysis: By analyzing the dependencies between data tables, a chain description is generated to explain how data flows between multiple tables. For example, data flows from the "Customer" table to the "Order" table through the "CustomerID" foreign key relationship.

[0101] Operation 22: Semantic analysis and textualization of business information.

[0102] According to the embodiments of this specification, business information may include tags, themes, layers, etc., which reflect the application of data tables in specific business scenarios and their backgrounds. The semantic analysis and textualization of this information are relatively independent of the table structure information, with the purpose of making the system more flexible in matching business semantics. By textualizing business information separately, it can be ensured that the system can filter and match according to the business background when processing complex queries.

[0103] Semantic parsing and textualization of business information may include the following steps, for example:

[0104] Tag and topic extraction: Extract the business attributes of a table through the business documents of the data table, the business hierarchical information of the data warehouse, or the business metadata. For example: "The table 'Customer' is classified under the 'Customer Management' topic and the tag is 'VIP Customer'."

[0105] Text processing: directly convert these business information into natural language descriptions to indicate the semantic meaning of the table in a specific business. For example: "Table 'Customer' is used to manage VIP customer information."

[0106] Operation 23. Semantic analysis and textualization of data quality.

[0107] According to the embodiments of this specification, data quality is an important factor affecting the accuracy of query results. Therefore, data quality information (such as indicators such as data accuracy, completeness, and timeliness) can be semantically parsed and textualized separately. In this way, the system can support filtering based on data quality during the selection process of candidate tables, ensuring that the generated database operation statements can return high-quality data results.

[0108] Semantic analysis and textualization of data quality may include the following steps:

[0109] Quality indicator extraction: Extract table-related quality information from data management tools or metadata management platforms, such as missing rate, duplication rate, data accuracy, etc. For example: "The data accuracy of table 'Customer' is 95%, and the missing rate is 2%."

[0110] Text processing: These indicators are converted into natural language to indicate the data quality level of the table. For example: "The data accuracy of table 'Customer' is 95%, and the data integrity is good.

[0111] Operation 24: Semantic parsing and textualization of management information.

[0112] According to the embodiments of this specification, the management information may include the life cycle, authority control, maintenance information, etc. of the data table. Such information has a certain reference value for the query results, but is not used as the main retrieval basis. Therefore, the management information can be used as a secondary reference factor.

[0113] The semantic analysis and textualization of management information may include the following steps:

[0114] Management information extraction: Extract the creation date, last update date, name or department information of the data table. For example: "Table 'Customer' was created in January 2020 and is maintained by the data management team."

[0115] Text processing: Convert management information into natural language descriptions to show the management dimensions of the table. For example: "Table 'Customer' is maintained by the data management team and updated monthly."

[0116] Operation 25: Semantic parsing and textualization of domain knowledge base.

[0117] According to the embodiments of this specification, in order to improve the accuracy of the system in business semantic judgment, a domain knowledge base can be introduced to assist in the parsing and matching of business semantics. The terms, rules and other information in the domain knowledge base will be semantically parsed and vectorized separately as a reference in business matching.

[0118] The semantic parsing and textualization of the domain knowledge base may include the following steps, for example:

[0119] Knowledge base information extraction: Extract relevant industry standards, business processes, etc. from the pre-built domain knowledge base. For example: "In the retail industry, VIP customers usually refer to customers who spend more than $10,000 per year."

[0120] Text processing: Textualize these domain knowledge to assist in judging business semantics. For example, "In the retail industry, VIP customers in the table 'Customer' refer to customers whose annual spending exceeds $10,000."

[0121] According to the embodiments of this specification, through the semantic parsing and textualization of such multidimensional information, the system can generate a comprehensive and detailed natural language description, thereby providing a rich semantic basis for subsequent intelligent table selection. These textual descriptions not only facilitate the system to understand the technical characteristics and business meaning of the data table in subsequent query processing, but also support flexible business semantic matching and filtering based on data quality. This processing method ensures that the system can comprehensively consider the multidimensional characteristics of the data table when generating database operation statements, thereby improving the accuracy and reliability of the generated results.

[0122] After obtaining the multi-dimensional vector information, the multi-dimensional vector information can be stored. In the "multi-dimensional vector information storage" stage, the core task of the system is to vectorize the textual data through a pre-trained model and store the generated vectors in the vector database so that the query request can be matched quickly and efficiently in the subsequent retrieval process.

[0123] According to an embodiment of the present specification, the textualized multidimensional data can be input into a pre-selected model (e.g., a sentence-transformers model, such as bge-large-zh-v1.5, etc.) for vectorization processing. The model will generate corresponding high-dimensional vectors based on the content of the text. These vectors represent the semantic information of the text and can be used for subsequent similarity retrieval. After obtaining the vectorized representation of the text, the system stores these vectors in a vector database. A vector database is a database specifically used to store and retrieve high-dimensional vectors. Exemplarily, the vector database may include FAISS, Milvus, etc. Optionally, not only the vector itself can be saved during storage, but also its original text information can be associated, which is convenient for backtracking during subsequent retrieval.

[0124] Multidimensional vectors can be stored, for example, in the following form:

[0125]

[0126] Among them, vector represents vector, metadata represents metadata, table represents data table, and fields represents fields.

[0127] In the "multi-dimensional vector information storage" phase, the model converts the textual data into vector representation, and stores these vectors in a special vector database. By vectorizing, storing and indexing the text information, the system can quickly retrieve and match the most relevant data table information in the subsequent query process, providing support for generating accurate database operation statements.

[0128] The method of retrieving the candidate list is described below in conjunction with another embodiment.

[0129] In this embodiment, the big model can be used to intelligently recommend the most suitable candidate table based on multi-dimensional information to meet the structural requirements and business scenarios of the target table. By performing semantic preprocessing and text processing on the target table and the candidate table, combined with multi-level vectorized retrieval, the system can generate a set of highly matched candidate tables. Then, using the constructed prompt word Prompt, this multi-dimensional information is passed to the big model, so that it can intelligently analyze and recommend the candidate table through natural language understanding capabilities. The whole process fully combines the multi-dimensional information of technical structure and business semantics to ensure the accuracy and reliability of candidate table recommendations.

[0130] The process of screening the candidate list may include, for example, the following operations 31 to 32:

[0131] Operation 31: perform semantic preprocessing on the target table.

[0132] According to an embodiment of the present specification, the table structure of the target table and its additional business information may be processed as text, so as to ensure that the core structure and semantic characteristics of the target table can be accurately captured in the subsequent vectorized search, and to promote matching with the existing data table. The text processing may include, for example:

[0133] Table structure analysis: The system can analyze the structural information of the target table, including table name, field name, data type, primary key and foreign key, etc. This information constitutes the core structure of the target table and is the basis for subsequent vectorized retrieval.

[0134] Additional business information analysis: In addition to structural information, a simple natural language description of the purpose of the target table or the target business is also provided. This helps to make more accurate matches based on business scenarios during vectorized search.

[0135] Text processing: The above table structure information and additional business information can be integrated into a natural language description to ensure that the text style is consistent with the existing data table in Chapter 1. This description not only contains the technical details of the table structure, but also describes the semantics of each field and the business background of the table in concise language, which is convenient for subsequent semantic retrieval.

[0136] The following is an example of a textual description of the target table:

[0137] Table name: Customer transaction record table (Customer_Transaction_Record)

[0138] Structure information, the table contains the following fields:

[0139] Transaction ID (Transaction_ID): Integer (INT), primary key, uniquely identifies each transaction.

[0140] Customer ID (Customer_ID): Integer (INT), foreign key, associated with the customer ID in the customer information table, used to identify the customer involved in the transaction.

[0141] Transaction_Amount: Floating point type (FLOAT), recording the amount of the transaction.

[0142] Transaction Date (Transaction_Date): Date type (DATE), records the specific date when the transaction occurs.

[0143] Payment method (Payment_Method): string type (VARCHAR), indicating the payment method used for the transaction, such as credit card, cash, etc.

[0144] Then, the above textual information can be vectorized and stored as technical structure vectors and business semantic vectors respectively for subsequent multi-dimensional retrieval.

[0145] Operation 32: multi-dimensional vector retrieval and sorting.

[0146] According to an embodiment of the present specification, operation 32 may include, for example, the following operations 321 to 323 .

[0147] Operation 321: Search based on different dimensions

[0148] According to an embodiment of the present specification, in this operation, the system may search the target table from multiple dimensions such as table name and description information, table structure and additional business information to ensure that the most matching candidate table can be found at both the technical level and the business level. The search results of each dimension are performed independently, and then these results are merged to provide a basis for subsequent weighting and sorting. Operation 321 may include operations 3211 to 3213, for example.

[0149] Operation 3211: Search based on table name and description information.

[0150] According to an embodiment of the present specification, the system can vectorize the table name of the target table and its accompanying natural language description to obtain a first retrieval vector. The similarity between the first retrieval vector and each table vector in the vector database, i.e., the first similarity, is calculated, and then the table vector in the vector database whose first similarity is greater than a first threshold, i.e., the first table vector, is determined, and the first data table corresponding to the first table vector is obtained. The table name, as a unique identifier of the target table, can provide a direct means of identification, while the natural language description includes a description of the business function, usage scenario, or other additional business information of the target table. This part of the retrieval focuses on finding data tables with a high degree of match from the perspective of business scenarios.

[0151] Operation 3212: search according to the table structure.

[0152] Exemplarily, in this embodiment, the system can use the DDL (Data Definition Language) structure of the target table as a retrieval basis. Based on this, retrieval based on the table structure may include, for example: the field name, data type, primary and foreign key relationship and other technical information of the target table are parsed and converted into a vector representation, i.e., a second retrieval vector. The second similarity between the second retrieval vector and each table vector in the vector database is calculated, and then the table vector whose second similarity is greater than the first threshold in the vector database, i.e., the second table vector, is determined, and the second data table corresponding to the second table vector is obtained. The system finds the candidate table that is closest to the DDL structure of the target table by calculating the similarity of the table structure at the field level and the table level. The system can focus on the semantic consistency between fields, the matching of field types, the order of fields and the grouping method, etc., to ensure a high degree of similarity in structure.

[0153] Operation 3213: Search according to the additional business information.

[0154] According to an embodiment of the present specification, for example, the additional business information can be vectorized to obtain a third retrieval vector. The third similarity between the third retrieval vector and each table vector in the vector database is calculated, and then the table vector in the vector database whose third similarity is greater than the first threshold, i.e., the third table vector, is determined, and the third data table corresponding to the third table vector is obtained. According to an embodiment of the present specification, in addition to the table name and structure information, the target table may also contain some additional business information, such as labels, domain knowledge, usage description, etc. This information is often not directly reflected by technical means, but it can provide strong support for business-level matching. For example, some tables may have clear application scenarios in specific business fields, and such scenarios rely on domain knowledge or specific business concepts. Through such processing, the system can find tables that are similar to the target table in business background or application scenario, ensuring that the final candidate table is not only technically matched, but also meets specific business needs.

[0155] By searching in three dimensions: table name and description information, table structure, and additional business information, the system can find more matching candidate tables at the technical and business levels. Each dimension focuses on different search priorities - table name and description information are mainly used for business semantic matching, table structure is used for technical matching, and additional business information is used to enhance the accuracy of business scenarios. This multi-dimensional search method can greatly improve the accuracy of candidate tables and lay a solid foundation for subsequent weighted sorting.

[0156] Operation 322: Score and sort the candidate list based on multiple dimensions.

[0157] In this operation, the system can score and sort the retrieved candidate tables to ensure that the selected candidate tables meet the requirements of the target table in all dimensions. By assigning different weights to each dimension and comprehensively scoring, the optimal candidate table, i.e., the target candidate table, is finally screened out. The core of this step is to merge the multi-dimensional retrieval results in the previous step, and weight the similarities of different dimensions to generate the final sorting results. Operation 322 may, for example, include operations 3221 to 3222.

[0158] Operation 3221, dimension merging.

[0159] In the previous operation, the system retrieved candidate tables based on multiple dimensions such as table name, description information, table structure, and additional business information. In this stage, the system can merge the search results of each dimension. Since the result set of each dimension may overlap, especially in the table name and table structure search, some tables will be retrieved in multiple dimensions, so it is necessary to deduplicate and merge the results.

[0160] According to an embodiment of the present specification, when the system merges candidate tables of different dimensions, it can perform deduplication processing on the table names to ensure that the same table does not appear repeatedly in the final candidate set. The deduplication processing is performed based on the table name, which can be obtained in the metadata information of the search result.

[0161] According to the embodiments of the present specification, for each retrieved candidate table, the system can record its similarity scores in different dimensions. For example, a table may have a high score in table structure similarity, but a low score in business information. The system can save these scores in the metadata of the candidate table for subsequent weighting.

[0162] Operation 3222: Weighted scoring.

[0163] According to an embodiment of the present specification, the combined candidate table can calculate a comprehensive score based on the scores of each dimension and the preset weights. The similarity score of each dimension will be multiplied by its corresponding weight to obtain a weighted score. The weights may include, for example, table structure similarity weights and business information similarity weights.

[0164] According to the embodiment of this specification, the table structure similarity is a key factor in evaluating whether the candidate table can technically meet the requirements of the target table, so the table structure similarity weight will be assigned a higher weight. By default, the weight of the table structure similarity can be set to 60%.

[0165] According to the embodiment of this specification, the business information similarity measures the matching degree of the candidate table at the business semantic level. Although the importance of business information depends on the specific application scenario, it has important reference value in a complex business environment, so the business information similarity weight can be given a weight of 40%.

[0166] According to the embodiments of this specification, in actual applications, the weight setting can be dynamically adjusted according to historical feedback and the needs of specific scenarios. The system can automatically adjust the weight distribution by analyzing the user's feedback data. For example, if in some business scenarios, the importance of business information is significantly higher than the table structure, the system can increase the weight of the business information.

[0167] According to the embodiment of this specification, the system will calculate the final score of each candidate table according to the following formula: Total score = (table structure score × 60%) + (business information score × 40%). For example, if the table structure similarity of a candidate table is 0.8 and the business information similarity is 0.7, then its total score is: Total score = (0.8 × 60%) + (0.7 × 40%) = 0.76. The higher the total score, the better the comprehensive performance of the candidate table in various dimensions.

[0168] Operation 323: sorting and filtering.

[0169] According to the embodiments of this specification, the system can sort all candidate tables according to their comprehensive scores. The candidate table with the highest comprehensive score will be ranked first. During the sorting process, a threshold can also be set to filter out candidate tables with scores lower than a certain set value to ensure the quality of the final candidate table.

[0170] According to the embodiments of this specification, the system can set a minimum score threshold according to the business scenario, and only tables with scores higher than the threshold will be used as valid candidate tables. For example, if the threshold set by the system is 0.7, then tables with a total score lower than 0.7 will be filtered out.

[0171] According to the embodiments of the present specification, after weighted scoring and threshold screening, the system can generate a final sorted list of candidate tables based on the total score. The tables in this list are highly matched with the target table in terms of both technical structure and business semantics, and have been weighted screened to give priority to meeting the most critical requirements.

[0172] According to the embodiments of this specification, by merging the search results of different dimensions and performing weighted scoring according to the weights, the system can provide a set of high-quality candidate tables for the target table. This multi-dimensional weighted scoring method can not only fully consider the similarities in technical structure, but also flexibly filter out tables that meet specific business needs through business information. Finally, the system generates a ranked list of candidate tables based on the comprehensive score to ensure that the subsequent large model can make choices based on accurate and comprehensive information.

[0173] In the following, another embodiment is used to illustrate how to select a target candidate table based on LLM. In this embodiment, a large language model (LLM) can be used to construct prompts (Prompts) that interact with the large model using the multi-dimensional information of the target table and the candidate table, so that the large model can intelligently select the candidate table that best meets the requirements, that is, the target candidate table. This process not only involves integrating the technical information and business information of the table into input suitable for processing by the large model, but also includes designing effective prompts to guide the model to make accurate candidate table recommendations.

[0174] Screening out the target candidate table based on the LLM may include, for example, operations 331 - 333 .

[0175] Operation 331: Obtain the target table information processed above and the retrieved candidate table information.

[0176] Operation 332: Combine input data and construct Prompt.

[0177] According to the embodiments of this specification, it is very important to construct an effective prompt when interacting with a large model. The prompt can contain all the key information of the target table and the candidate table, and clearly express the task of the model, that is, to select the most matching table from the candidate table based on the table structure and business background information.

[0178] Exemplarily, in this embodiment, Prompt may include:

[0179] Target table information:

[0180] Table name: Customer transaction record table

[0181] Field information:

[0182] -Transaction ID (Transaction_ID): integer, primary key.

[0183] -Customer ID (Customer_ID): integer, foreign key.

[0184] -Transaction_Amount: floating point type.

[0185] -Transaction_Date: Date type.

[0186] -Payment_Method: string.

[0187] Business Background:

[0188] -This form is used to analyze customers' consumption behavior and payment preferences.

[0189] -Data is needed for monthly report generation by the finance department and customer analysis by the marketing department.

[0190] Candidate list information:

[0191] Table name: Candidate table 1

[0192] Field information:

[0193] -...

[0194] -...

[0195] Task: Based on the above information, determine which candidate table is most suitable as the source table to help fill the target table with data. Please analyze based on the table structure, business background, and data relationships.

[0196] Operation 333, LLM processing and recommended output

[0197] According to the embodiments of the present specification, after receiving the input prompt, LLM can parse various information of the target table and the candidate table through its powerful natural language understanding ability, including the similarity of table structures, matching degree of business semantics, and potential data relationships.

[0198] LLM can analyze the following dimensions based on the context provided:

[0199] Structural similarity: LLM can analyze the structural similarity between the target table and the candidate table to determine whether their field types, primary and foreign key relationships, etc. match.

[0200] Business semantic matching: LLM can also determine whether the candidate table meets the business requirements of the target table based on business background information. For example, if the target table involves "sales order" and the candidate table is "customer information", the model will determine their relevance based on business semantics.

[0201] Comprehensive consideration: LLM can make a comprehensive judgment based on technical information (table structure) and business information (business semantics) to provide recommended candidate tables.

[0202] In the process of recommending candidate lists, the output candidate list may not fully meet the user's needs. In order to ensure that the final output meets the user's requirements, a multi-round dialogue process can be introduced, and the user can provide feedback and supplement information based on the output results to gradually optimize the candidate list recommendation process.

[0203] For example, a multi-turn conversation flow could include:

[0204] First recommendation: LLM outputs candidate table A, but the user finds that the table is missing an important field.

[0205] User feedback: "The payment method field is missing in candidate table A. Please recommend another candidate table."

[0206] Second recommendation: LLM readjusts the screening criteria based on the feedback information and outputs candidate list B.

[0207] The user gave another feedback: "The data quality of candidate table B does not meet the requirements, and the data missing rate is too high."

[0208] Final recommendation: Based on the feedback information and supplementary requirements, LLM screened out candidate table C, whose structure and data quality met the requirements, and the user accepted the recommendation result.

[0209] Through this multi-round dialogue mechanism, users can continuously supplement business and technical requirements based on actual needs, and guide the large model to gradually optimize the selected candidate tables until the best table that meets business goals is recommended. This mechanism ensures that the system can handle complex business needs and improve the accuracy of candidate table recommendations.

[0210] Finally, LLM will output a recommended candidate table. Optionally, LLM can also output an explanation with the basis for the recommendation. This recommendation basis can help developers understand the decision-making process of the model. Through the intelligent recommendation of the large model, the system can more efficiently find the candidate table that best meets the requirements of the target table, thereby simplifying the complexity of database operation statement generation and improving the accuracy of data governance.

[0211] By introducing LLM's intelligent recommendation of candidate tables, the candidate table screening and recommendation process is more efficient. With the help of LLM's natural language processing and deep semantic understanding capabilities, the system can make more intelligent decisions based on the similarity of table structures and business background. This not only improves the accuracy of matching, but also greatly reduces the intervention of manual rules, and enhances the flexibility and automation capabilities of the entire data governance process. Ultimately, the intelligently recommended candidate tables will serve as the basis for the generation of subsequent database operation statements, further simplifying the complex operation processes in data governance and improving the overall efficiency of the system.

[0212] The following describes a method for generating database operation statements using LLM in conjunction with another embodiment.

[0213] According to an embodiment of the present specification, by inputting the structural information of the target table and the candidate table into the model, data migration and governance database operation statements from the candidate table to the target table are automatically generated. During the generation process, the system can automatically verify the correctness of the database operation statement, and make adjustments through multiple rounds of dialogue when necessary to ensure that the final output meets the requirements. In addition, complex operations such as field matching, type conversion, and business logic conversion can also be handled intelligently, greatly improving the automation and accuracy of data governance. The method of generating database operation statements using LLM may, for example, include the following operations 341 to 343.

[0214] Operation 341: Generate a context according to the target table information and the candidate table information.

[0215] According to an embodiment of the present specification, the target table information may include its DDL (Data Definition Language) description and business context. DDL defines technical information such as the target table's fields, data types, primary keys, and foreign key relationships, while the business context provides the table's application scenarios, uses, and roles in the business process. By integrating this information, the model can understand the data format and business requirements expected by the target table.

[0216] The candidate table information comes from the multi-dimensional retrieval and recommendation process in the previous chapter. The DDL structure and business semantics of each candidate table also need to be considered in combination with the target table. The model needs to fully grasp the information of these candidate tables to ensure that the generated database operation statements can correctly extract the data in the candidate table and match the target table structure.

[0217] Based on the target table and candidate table information, a context is constructed for generating database operation statements. In order for the large model to generate effective database operation statements, a complete context must be provided for it. This context includes not only the structure information (DDL) of the target table and the DDL of the candidate table, but also business information to help the model understand the logical relationship between tables.

[0218] Operation 342: Generate a Prompt.

[0219] The design of prompt is the core of large model task execution. In the scenario of database operation statement generation, the instruction must be clear and concise to ensure that the model can correctly understand the task objectives. The core task of this instruction is to let the model extract data from the candidate table based on the context and insert it into the target table.

[0220] The prompt should clarify the following aspects:

[0221] (1) Task: Instruct the big model to generate database operation statements for migrating data from the candidate table to the target table, and ensure that the fields, data types, etc. match.

[0222] (2) Technical requirements: When generating database operation statements, you must consider the matching of field types, the maintenance of primary and foreign key relationships, and possible data conversion logic. Field matching: The model needs to determine whether the field types of the target table and the candidate table are consistent and whether type conversion is required. Primary and foreign key maintenance: Ensure that the data inserted into the candidate table does not violate the primary and foreign key constraints of the target table. Data conversion logic: For example, operations such as amount format conversion and date format standardization.

[0223] (3) Business relevance: Database operation statements must be based on the business needs of the target table to ensure that the data in the candidate table meets its business semantic requirements.

[0224] By designing concise and clear instructions, the accuracy of model generation can be significantly improved.

[0225] Operation 343: Generate and automatically verify database operation statements.

[0226] After the big model receives the context and prompt, it will generate a database operation statement based on the DDL and business information of the target table and candidate table. The generated database operation statement should be able to extract data from the candidate table and correctly insert it into the target table, ensuring the consistency of technical information such as fields, data types, primary and foreign keys.

[0227] After generating the database operation statement, the system can automatically verify the database operation statement to ensure that it can be successfully executed in the database environment. The verification content includes:

[0228] (1) Field matching: Verify whether the fields in the candidate table and the target table match, whether the data types are consistent, and whether data conversion is required. If the field names are exactly the same, mapping is performed directly; if the field names are inconsistent but the types are the same, the system attempts to automatically identify them based on the context. For example: the amount field in the target table matches the transaction_amount field in the candidate table.

[0229] (2) Data conversion logic: Ensure that the conversion logic of key fields such as amount and date is correct. If there are specific requirements in the business, the model will perform data conversion according to business rules. For example, convert the amount field according to the exchange rate:

[0230] (3) Primary key and foreign key relationships: Ensure that the inserted data does not destroy the integrity and constraints of the target table.

[0231] If the generated database operation statement fails to pass the verification or does not meet the business requirements, the system will automatically enter the multi-round dialogue mode. By interacting with the user, the system will prompt the user to supplement the missing business information or technical details, such as field mapping rules, data type conversion requirements, etc. The large model generates a new database operation statement based on the new input and performs automatic verification again. This process will be repeated until the system generates a database operation statement that meets the requirements.

[0232] Operation 344: Output and execute database operation statements.

[0233] The verified database operation statements will be used as the final database operation statements for data governance and executed by the data engineering team. These database operation statements have not only passed the automatic generation and verification of the model, but have also been adjusted and optimized multiple times based on business needs to ensure that the final database operation statements can meet the data loading requirements of the target table.

[0234] The generation of database operation statements based on LLM fully utilizes the advantages of large models in natural language processing and semantic understanding, and can automatically generate efficient and accurate database operation statements based on the technical structure and business semantics of the target table and candidate table. Through the automated verification and validation mechanism, it not only improves the efficiency of the data governance process, but also enhances the flexibility and accuracy of generating database operation statements. Ultimately, the system can provide a set of intelligent and automated database operation statement generation processes to simplify complex operations in data management and ensure the high quality and consistency of data processing.

[0235] In addition, through intelligent table selection and database operation statement generation technology, the data management process from source table to target table can be automated. Users can automatically match the target table and source table while generating database operation statements, complete the cleaning, conversion and management operations during the data flow process, and greatly improve the efficiency and accuracy of data management.

[0236] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device for generating a database operation statement and a terminal used therein.

[0237] The embodiments of the device for generating database operation statements in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 3 The figure is a hardware structure diagram of the computer device where the database operation statement generation device of the embodiment of this specification is located, except Figure 3In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the server or electronic device where the device 331 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0238] like Figure 4 As shown, Figure 4 : is a block diagram of a device for generating a database operation statement according to an exemplary embodiment of the present specification, the device comprising:

[0239] A search module 410 is used to perform similarity search in a vector database based on multi-dimensional information of a target table to obtain a set of candidate tables;

[0240] A first prompt word determination module 420, configured to determine a first prompt word for screening candidate tables according to the table structure and business semantics of each candidate table in the candidate table set and the table structure and business semantics of the target table;

[0241] A screening module 430 is used to screen a target candidate list from the candidate list set according to the first prompt word using a large language model;

[0242] A second prompt word determination module 440 is used to determine a second prompt word for generating a database operation statement according to the data definition language and business semantics of the target table and the table structure and business semantics of the target candidate table, wherein the database operation statement is used to insert the data in the target candidate table into the target table;

[0243] The generating module 450 is used to generate a database operation statement according to the second prompt word by using the large language model.

[0244] Optionally, the multidimensional information may include table name, description information, table structure and additional business information; the retrieval module may include:

[0245] A first vectorization submodule, used for vectorizing the table name and description information to obtain a first search vector;

[0246] A first similarity calculation submodule, used to calculate a first similarity between the first search vector and each table vector in the vector database;

[0247] A first table vector determination submodule, used to determine a first table vector in a vector database whose first similarity is greater than a first threshold, the first table vector corresponding to a first data table;

[0248] A second vectorization submodule is used to vectorize the table structure to obtain a second search vector;

[0249] A second similarity calculation submodule, used to calculate a second similarity between the second search vector and each table vector in the vector database;

[0250] A second table vector determination submodule, used to determine a second table vector in the vector database whose second similarity is greater than the first threshold, the second table vector corresponding to the second data table;

[0251] A third vectorization submodule, used for vectorizing the additional service information to obtain a third search vector;

[0252] A third similarity calculation submodule, used to calculate a third similarity between the third search vector and each table vector in the vector database;

[0253] A third table vector determination submodule, used to determine a third table vector in the vector database whose third similarity is greater than the first threshold, the third table vector corresponding to the third data table;

[0254] The merging submodule is used to merge the first data table, the second data table and the third data table to obtain a candidate table set.

[0255] Optionally, the device may further include:

[0256] A scoring module, for performing weighted summation of the first similarity, the second similarity and the third similarity of each candidate table in the candidate table set to obtain a comprehensive score of the data table;

[0257] The deletion module is used to delete the candidate tables whose comprehensive scores are lower than the score threshold in the candidate table set.

[0258] Optionally, the first prompt word may include target table information, candidate table information and task information, wherein the task information is used to instruct the large model to analyze the candidate table information based on the table structure, business semantics and numerical relationship, so as to screen out the target candidate table with the highest matching degree with the target table information;

[0259] The first prompt word determination module may include:

[0260] A first embedding submodule, used to embed the table structure and business semantics of the target table into the target table information in the first prompt word;

[0261] The second embedding submodule is used to embed the table structure and business semantics of each candidate table into the candidate table information in the first prompt word.

[0262] Optionally, the device may further include:

[0263] A receiving module, used for receiving feedback information on the target candidate list;

[0264] A correction module, used to generate correction prompt words according to feedback information;

[0265] The re-screening module is used to use the large language model to screen out a new target candidate list from the candidate list set according to the revised prompt words.

[0266] Optionally, the second prompt word may include task information, technical requirement information and business association information; the task information is used to instruct the big model to generate database operation statements based on the target table information and the target candidate table information; the technical requirement information is used to indicate the technical requirements that the big model should meet when generating database operation statements, and the technical requirements include field matching requirements, primary and foreign key constraint requirements and data conversion logic requirements; the business association information is used to indicate the business semantic requirements that the big model should meet when generating database operation statements.

[0267] Optionally, the device may further include:

[0268] The verification module is used to verify the database operation statement, and the verification includes at least one of the following operations:

[0269] Verify that when executing database operation statements, the data inserted into the target table matches the original fields of the target table;

[0270] Verify whether the data conversion logic in the database operation statement is correct;

[0271] Verify whether the data inserted into the target table destroys the integrity and primary and foreign key constraints of the target table when the database operation statement is executed.

[0272] According to the embodiments of the present specification, through natural language processing technology combined with multi-dimensional vector retrieval technology, the user's natural language query can be automatically converted into database operation statements, and the appropriate data table can be intelligently selected, thereby greatly reducing the user's understanding requirements for the database and business logic, allowing non-technical users to perform complex data queries efficiently.

[0273] Correspondingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: perform similarity search in a vector database based on multi-dimensional information of a target table to obtain a set of candidate tables; determine a first prompt word for screening candidate tables according to the table structure and business semantics of each candidate table in the set of candidate tables, as well as the table structure and business semantics of the target table; use a large language model to screen out a target candidate table from the set of candidate tables according to the first prompt word; determine a second prompt word for generating a database operation statement according to the data definition language and business semantics of the target table, as well as the table structure and business semantics of the target candidate table; and use the large language model to generate a database operation statement according to the second prompt word.

[0274] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0275] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0276] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0277] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0278] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0279] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for generating a database operation statement, characterized in that: The method comprises: Based on the multi-dimensional information of the target table, similarity search is performed in the vector database to obtain a set of candidate tables; Determining, according to the table structure and business semantics of each candidate table in the candidate table set and the table structure and business semantics of the target table, a first prompt word for screening candidate tables according to the table structure and business semantics; Using a large language model, a target candidate list is screened out from the candidate list set according to the first prompt word; Determining, according to the data definition language and business semantics of the target table and the table structure and business semantics of the target candidate table, a second prompt word for generating a database operation statement, wherein the database operation statement is used to insert the data in the target candidate table into the target table; A database operation statement is generated according to the second prompt word by using a large language model.

2. The method according to claim 1, characterized in that The multidimensional information includes table name, description information, table structure and additional business information; the multidimensional information based on the target table performs similarity search in the vector database to obtain a candidate table set, including: Vectorize the table name and description information to obtain a first search vector; Calculating a first similarity between the first search vector and each table vector in the vector database; Determine a first table vector in the vector database whose first similarity is greater than a first threshold, wherein the first table vector corresponds to a first data table; Vectorizing the table structure to obtain a second search vector; Calculating a second similarity between the second search vector and each table vector in the vector database; Determine a second table vector in the vector database whose second similarity is greater than the first threshold, wherein the second table vector corresponds to a second data table; Vectorizing the additional service information to obtain a third search vector; Calculating a third similarity between the third search vector and each table vector in the vector database; Determine a third table vector in the vector database whose third similarity is greater than a first threshold, wherein the third table vector corresponds to a third data table; The first data table, the second data table and the third data table are merged to obtain the candidate table set.

3. The method according to claim 2, characterized in that The method further comprises: For each candidate table in the candidate table set, weighted sum of the first similarity, the second similarity and the third similarity of the candidate table is performed to obtain a comprehensive score of the data table; Delete the candidate tables in the candidate table set whose comprehensive scores are lower than the score threshold.

4. The method according to claim 1, characterized in that: The first prompt word includes target table information, candidate table information and task information, wherein the task information is used to instruct the large model to analyze the candidate table information based on table structure, business semantics and numerical relationship, so as to screen out the target candidate table with the highest matching degree with the target table information; The step of determining, based on the table structure and business semantics of each candidate table in the candidate table set and the table structure and business semantics of the target table, a first prompt word for screening the candidate tables according to the table structure and business semantics includes: Embed the table structure and business semantics of the target table into the target table information in the first prompt word; The table structure and business semantics of each candidate table are embedded in the candidate table information in the first prompt word.

5. The method according to claim 4, characterized in that The method further comprises: receiving feedback information on the target candidate list; Generating a revised prompt word according to the feedback information; A new target candidate list is screened out from the candidate list set according to the modified prompt word by using a large language model.

6. The method according to claim 1, characterized in that The second prompt word includes task information, technical requirement information and business association information; the task information is used to instruct the big model to generate the database operation statement based on the target table information and the target candidate table information; the technical requirement information is used to indicate the technical requirements that the big model should meet when generating the database operation statement, and the technical requirements include field matching requirements, primary and foreign key constraint requirements and data conversion logic requirements; The business association information is used to indicate the business semantic requirements that the large model should meet when generating the database operation statement.

7. The method according to claim 1, characterized in that The method further comprises: Verifying the database operation statement, wherein the verification includes at least one of the following operations: Verifying whether the data inserted into the target table matches the original fields of the target table when the database operation statement is executed; Verify whether the data conversion logic in the database operation statement is correct; Verify whether the data inserted into the target table destroys the integrity and primary and foreign key constraints of the target table when the database operation statement is executed.

8. A device for generating a database operation statement, characterized in that: The device comprises: A retrieval module is used to perform similarity retrieval in a vector database based on multi-dimensional information of a target table to obtain a set of candidate tables; A first prompt word determination module, used to determine a first prompt word for screening candidate tables according to the table structure and business semantics of each candidate table in the candidate table set and the table structure and business semantics of the target table; A screening module, configured to screen a target candidate list from the candidate list set according to the first prompt word by using a large language model; a second prompt word determination module, configured to determine a second prompt word for generating a database operation statement according to the data definition language and business semantics of the target table and the table structure and business semantics of the target candidate table, wherein the database operation statement is used to insert the data in the target candidate table into the target table; A generation module is used to generate a database operation statement according to the second prompt word by using a large language model.

9. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images