Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment. The method comprises the following steps: acquiring a business demand input by a target object; a historical reference sample corresponding to the business demand is determined, the historical reference sample comprises a historical demand and historical query content, and the historical query content comprises a historical table name and / or a historical field name; generating a cue word corresponding to each historical reference sample according to the cue word template, and generating an output result of the cue word by adopting a large model; and determining reply content corresponding to the business demand according to the historical reference sample and the output result. According to the method and the device, the technical problem of low SQL statement generation accuracy caused by difficulty in accurately extracting related information from a large model when the large model processes a complex database is solved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to a data processing method, apparatus, and electronic device. Background Art

[0002] With the continuous development of natural language processing technology, NL2SQL has become an important direction for large model applications, which allows users to generate SQL query statements through natural language descriptions. In an NL2SQL system, the following several links are usually included: 1. determining the scope of tables and fields; 2. converting natural language into SQL; 3. correcting the generated SQL. Among them, the first part, that is, the table and field selection module, is the key part determining the accuracy of SQL generation. However, there are many limitations in the SQL generation methods in related technologies. Some systems require users to manually fill in the required table or field information, but users are not familiar with this. There are also some systems that try to include all table or field information in the prompt, which not only makes the SQL generation process cumbersome but also increases the possibility of errors. Especially when dealing with complex databases, NL2SQL models often have difficulty accurately extracting relevant information from them, resulting in the failure of SQL statement generation.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of this application provide a data processing method, apparatus, and electronic device to at least solve the technical problem that when a large model processes a complex database, it is difficult to accurately extract relevant information from it, resulting in a low accuracy rate of SQL statement generation.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, including: obtaining a business requirement input by a target object; determining a historical reference example corresponding to the business requirement, where the historical reference example includes a historical requirement and historical query content, and the historical query content includes a historical table name and / or a historical field name; generating a prompt corresponding to each historical reference example according to a prompt template, and using a large model to generate an output result of the prompt; determining a reply content corresponding to the business requirement according to the historical reference example and the output result.

[0006] Optionally, the method further includes: when the business requirement is a table query, the historical query content includes a historical table name; when the business requirement is a field query, the historical query content includes a historical table name and a historical field name.

[0007] Optionally, generate prompts corresponding to each historical reference example according to the prompt template, and use the large model to generate the output results of the prompts, including: in the case where the business requirement is table query, determine the first historical reference example corresponding to the business requirement, where the first historical reference example includes the historical requirement and the historical table name; generate the first prompt corresponding to each first historical reference example according to the first prompt template, and use the large model to generate the first output result of the first prompt, where the first output result includes the reason for selecting the table and the set of selected tables.

[0008] Optionally, determine the response content corresponding to the business requirement based on the historical reference example and the output result, including: in the case where the output result is the first output result, obtain the historical table set composed of the historical table names in the first historical reference example; merge the historical table set and the set of selected tables in the first output result to obtain the first union; after de-duplicating the first union, obtain the first response content corresponding to the business requirement.

[0009] Optionally, generate prompts corresponding to each historical reference example according to the prompt template, and use the large model to generate the output results of the prompts, including: in the case where the business requirement is field query, determine the second historical reference example corresponding to the business requirement, where the second historical reference example includes the historical requirement, the historical table name, and the historical field name, and the historical table name is the table name corresponding to the set of tables selected in the first output result generated by the large model when performing table query on the business requirement; generate the second prompt corresponding to each second historical reference example according to the second prompt template, and use the large model to generate the second output result of the second prompt, where the second output result includes the reason for selecting the field and the set of selected fields.

[0010] Optionally, determine the response content corresponding to the business requirement based on the historical reference example and the output result, including: in the case where the output result is the second output result, obtain the historical field set composed of the historical field names in the second historical reference example; merge the historical field set and the set of selected fields in the second output result to obtain the second union; after de-duplicating the second union, obtain the second response content corresponding to the business requirement.

[0011] Optionally, the vector database where the historical reference examples are located is determined as follows: Obtain the historical retrieval records stored locally; verify the historical retrieval records to obtain historical question pairs, where the historical question pairs include historical questions, historical table information, and historical field information; use a vectorization model to vectorize the historical questions to obtain historical question vectors; store the historical question vectors, historical questions, historical table information, and historical field information in the vector database; or, obtain the historical query results corresponding to the historical questions through an SQL statement generation framework, where the historical query results include historical table information and historical field information; in the case where the historical query results are normal data, determine the historical query results and the historical questions as historical question pairs; use a vectorization model to vectorize the historical questions to obtain historical question vectors; store the historical question vectors, historical questions, and historical query results in the vector database.

[0012] According to another aspect of the embodiments of the present application, there is also provided a data processing device, including: an acquisition module, configured to acquire the business requirements input by the target object; a first determination module, configured to determine the historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents; a generation module, configured to generate a prompt corresponding to each historical reference example according to a prompt template, and use a large model to generate the output result of the prompt; a second determination module, configured to determine the reply content corresponding to the business requirements according to the historical reference examples and the output result.

[0013] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory, configured to store program instructions; a processor, connected to the memory, configured to execute the program instructions to implement the following functions: acquire the business requirements input by the target object; determine the historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents; generate a prompt corresponding to each historical reference example according to a prompt template, and use a large model to generate the output result of the prompt; determine the reply content corresponding to the business requirements according to the historical reference examples and the output result.

[0014] According to still another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, where the non-volatile storage medium includes a stored computer program, and the device where the non-volatile storage medium is located executes the above data processing method by running the computer program.

[0015] In an embodiment of the present application, by obtaining the business requirements input by the target object; determining the historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents, and the historical query contents include historical table names and / or historical field names; generating prompts corresponding to each historical reference example according to the prompt templates, and using a large model to generate the output results of the prompts; determining the response content corresponding to the business requirements according to the historical reference examples and the output results, the purpose of accurately identifying and screening relevant tables and fields in the database is achieved, thereby realizing the technical effect of improving the NL2SQL conversion accuracy rate, and further solving the technical problem that when the large model processes a complex database, it is difficult to accurately extract relevant information from it, resulting in a low accuracy rate of SQL statement generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0017] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of a data processing method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of obtaining a first historical reference example according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a first prompt template according to an embodiment of the present application;

[0021] Figure 5a is a schematic diagram of a first prompt a according to an embodiment of the present application;

[0022] Figure 5b is a schematic diagram of a first prompt b according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of obtaining a first output result according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of obtaining a second historical reference example according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of a second prompt template according to an embodiment of the present application;

[0026] Figure 9aSchematic diagram of a second prompt a according to an embodiment of the present application;

[0027] Figure 9b Schematic diagram of a second prompt b according to an embodiment of the present application;

[0028] Figure 10 Schematic diagram of obtaining a second output result according to an embodiment of the present application;

[0029] Figure 11 Schematic diagram of an input sample according to an embodiment of the present application;

[0030] Figure 12 Schematic diagram of an output sample according to an embodiment of the present application;

[0031] Figure 13 Schematic diagram of result storage according to an embodiment of the present application;

[0032] Figure 14 Structural diagram of a data processing device according to an embodiment of the present application. Detailed implementation manners

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0035] The information collected in the embodiments of this application is information and data authorized by the user or fully authorized by all parties. Moreover, for the processing of relevant data such as collection, storage, use, processing, transmission, provision, disclosure, and application, all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures are taken, it does not violate public order and good customs, and a corresponding operation entry is provided for the user to choose to authorize or reject the automated decision result; if the user chooses to reject, the expert decision-making process will be entered.

[0036] First, some nouns or terms that appear in the process of explaining the embodiments of this application are applicable to the following explanations:

[0037] NL2SQL (Natural Language to SQL) large model: It is an advanced algorithm model that uses natural language processing (NLP) technology to convert human natural language input into SQL (Structured Query Language) query statements. Its purpose is to simplify the database query process, enabling users without a technical background to access and operate the database through intuitive natural language descriptions without having to master complex SQL syntax. The NL2SQL large model is usually based on deep learning technologies such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), attention mechanisms, and more advanced Transformer architectures. These models are trained on large-scale database query statements and their corresponding natural language descriptions, and can understand the semantics and intentions in natural language and accurately convert them into SQL statements for performing data retrieval, analysis, or operations.

[0038] In the related art, when dealing with complex databases, the NL2SQL model often has difficulty accurately extracting relevant information from them, resulting in the failure of SQL statement generation. This technical problem highlights the deficiencies of the existing technology in processing large-scale table / field information. To solve this problem, some new methods have been proposed in the related art, such as using keyword information to expand synonyms and then finding approximate fields as candidates. However, such methods are often not applicable in actual production because there are a large number of abbreviations and shorthands in the table field information in actual production.

[0039] To solve the problems existing in the related art, the embodiments of this application provide a data processing method, which can run on Figure 1 the computer terminal shown below, and the following is an explanation of this computer terminal.

[0040] The data processing method embodiments provided by the embodiments of this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. As Figure 1 shown, the computer terminal 10 may include one or more processors (the processors may include, but are not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA, shown as 102a, 102b, ……, 102n in the figure), a memory 104 for storing data, and a transmission module 106 for communication functions connected by wired and / or wireless networks. In addition, it may further include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown in Figure 1 or have a different configuration from

[0041] It should be noted that the above one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this article. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10. As involved in the embodiments of the present application, the data processing circuit is a processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor, and these remote memories may be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission module 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer terminal 10.

[0045] It should be noted here that, in some alternative embodiments, the above Figure 1 illustrated computer terminal may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 1 is only an example of a specific specific instance and is intended to illustrate the types of components that may exist in the above computer terminal.

[0046] Under the above operating environment, an embodiment of a data processing method is provided in an embodiment of the present application. It should be noted that the steps illustrated in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is illustrated in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0047] Figure 2 is a flowchart of a data processing method according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0048] Step S202, obtaining the service requirements input by the target object.

[0049] In the above step S202, in scenarios such as data query and analysis, the target object refers to the user using the system, and the service requirement is the specific information or operation that the user hopes to obtain from the system.

[0050] In some embodiments of the present application, the target object can input business requirements through an interface or a text box. For example, the business requirement can be "display the products with the highest sales in the first quarter of 2023", etc. After receiving the business requirements input by the target object, the natural language understanding module in the system will parse the above business requirements, including identifying user intentions, keywords, time ranges, data metrics, etc., so as to understand the essence of the business requirements.

[0051] Step S204, determine the historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents, and the historical query contents include historical table names and / or historical field names.

[0052] In the above step S204, historical reference examples related or similar to the business requirements can be determined from the vector database to more accurately predict the database tables and fields involved in the current business requirements, so as to generate more accurate SQL statements. The historical reference examples contain the query requirements (historical requirements) put forward by users in the past and the historical query contents involved in the SQL query statements corresponding to these historical requirements. The historical query contents include specific table names (historical table names) and / or field names (historical field names). By analyzing these examples, the system can discover common patterns of specific types of queries and associated database tables and fields, thereby improving the understanding and processing capabilities of new queries.

[0053] Step S206, generate prompts corresponding to each historical reference example according to the prompt template, and use the large model to generate the output results of the prompts.

[0054] In the above step S206, the prompt template (Prompt Templates) is a specially designed text structure used to guide and inspire the large model to generate more targeted and accurate outputs. In the NL2SQL (Natural Language to SQL) scenario, the prompt template combines specific historical reference examples to help the large model understand the requirements and context of the current query, so as to more accurately recommend table names and field names.

[0055] When the system determines the historical reference examples related to the current business requirements, it will fill the key information of the historical reference examples (such as historical requirement descriptions, historical table names, and historical field names) into the specially designed prompt template to generate a series of prompts. The generated prompts will then be submitted to the large model. Based on its understanding ability trained on a large amount of data, the large model analyzes the prompts, identifies the patterns and associations therein, and generates output results. The output results usually include the table names and / or field names recommended for the current query, as well as the basis or reasons for selecting these tables and / or fields.

[0056] Step S208: Determine the response content corresponding to the business requirement based on the historical reference examples and the output result.

[0057] In the above step S208, since the historical reference examples include the historical table names and / or historical field names corresponding to the business requirement, and the output result includes the recommended table names and / or field names, according to the union of the data in the historical reference examples and the output result, for example, performing a union operation on the historical table names in the historical reference examples and the recommended table names in the output result, and / or, performing a union operation on the historical field names in the historical reference examples and the recommended field names in the output result, the response content corresponding to the business requirement can be obtained. After obtaining the response content, generate an SQL statement based on this response content.

[0058] Through the above steps, the purpose of accurately identifying and filtering relevant tables and fields in the database is achieved, thus realizing the technical effect of improving the accuracy of NL2SQL conversion. Furthermore, it solves the technical problem that when the large model processes a complex database, it is difficult to accurately extract relevant information from it, resulting in a low accuracy rate of SQL statement generation. The following is an explanation.

[0059] In the above data processing method, the method further includes: when the business requirement is a table query, the historical query content includes the historical table name; when the business requirement is a field query, the historical query content includes the historical table name and the historical field name.

[0060] In some embodiments of the present application, when the user's business requirement focuses on querying a specific table or understanding the data of interest, the historical query content mainly focuses on the historical table name. For example, if the user asks "Query the information of all employees", the system will mainly search for historical reference examples in the history that involve the "Employee Information" table to determine the most relevant database table. If the user's requirement focuses on querying specific fields or understanding which fields in a certain table are relevant, at this time, the historical query content will include the historical table name and the historical field name. The system not only needs to consider the table involved in the historical query, but also analyze the specific fields involved in the queried table. For example, when the user says "Please list the salary and start date of all employees", the system will search for query examples in the history that are associated with the "Employee" table and contain the fields of "salary" and "start date".

[0061] In step S206 of the above data processing method, a prompt is generated for each historical reference example according to the prompt template, and the output result of the prompt is generated using a large model, including: in the case where the business requirement is table query, determining a first historical reference example corresponding to the business requirement, where the first historical reference example includes a historical requirement and a historical table name; generating a first prompt for each first historical reference example according to the first prompt template, and generating a first output result of the first prompt using a large model, where the first output result includes the reason for selecting the table and the set of selected tables.

[0062] In some embodiments of the present application, the system first identifies that the business requirement proposed by the user includes table query. Based on the nature of the user requirement, the system filters historical examples related to table query from the vector database, and the historical examples include historical requirement descriptions and historical table names involved in historical queries. According to the semantic similarity between the current requirement and the historical examples, the most relevant set of first historical reference examples is selected as a reference. To guide the large model to generate information related to table query, a first prompt is generated for each first historical reference example through the first prompt template. The first prompt template contains necessary fields to enable the large model to understand the user's intention and how to recommend table names based on the first historical reference example. The generated first prompt is submitted to the large model for prediction, and the large model generates a first output result based on the information in the first prompt, combined with the patterns and associations it has learned from a large amount of data. The first output result includes the reason for selecting the table and the set of selected tables, and is returned in JSON format.

[0063] Through the above process, the system can intelligently recommend database tables based on the specific requirements of the user, using historical reference examples and the prediction ability of the large model, greatly simplifying the user's operation process and improving the query efficiency and accuracy. This method is especially suitable for the situation where the user is not familiar with the database structure or the database contains a large number of tables, and can effectively narrow the search scope and provide accurate query suggestions.

[0064] In step S208 of the above data processing method, based on the historical reference example and the output result, a response content corresponding to the business requirement is determined, including: in the case where the output result is the first output result, obtaining a historical table set composed of the historical table names in the first historical reference example; merging the historical table set and the set of selected tables in the first output result to obtain a first union; and after de-duplicating the first union, obtaining a first response content corresponding to the business requirement.

[0065] In some embodiments of the present application, first, the table names involved in the history are extracted from the selected first historical reference example to obtain a historical table set. The historical table set is based on all the table names in the first historical reference example that is most similar to the current business requirement found by the system through vectorized retrieval technology. The historical table set is merged with the table set (or recommended table set) selected from the first output result generated by the large model. The first output result includes the reasons for selecting the tables and the selected table set. The merging operation is to merge all the table names in the historical table set and the table set selected from the first output result. To ensure the accuracy and conciseness of the reply content, the system will perform deduplication on the merged first union set, remove duplicate table names, and retain only the uniquely occurring entries. The first union set after deduplication contains all the database tables related to the current business requirement. The system integrates the deduplicated table set into the first reply content.

[0066] The process of table query is described in detail below in combination with a specific process, including the following steps:

[0067] 1) According to the user's business requirement q, p1 first historical reference examples Aw1 are retrieved from the vector database through a vectorized model. Aw1 includes {historical requirement, historical table name}. Let the historical table names in Aw1 be the historical table set T1;

[0068] 2) Use the first prompt template to generate p1 first prompts in combination with each first historical reference example;

[0069] 3) Call the semantic large model one by one for the p1 first prompts. Each first prompt obtains n1 output results Aw2 (n1≥1). Aw2 includes {reasons for selecting the tables, selected table set}. After the p1 first prompts are executed, p1*n1 results Aw2 are obtained, that is, the first output result is obtained. Let the selected table set in Aw2 be T2;

[0070] 4) Take the union of T1 and T2 as the final first reply content Tr.

[0071] The above process can be illustrated by the following example. For example:

[0072] 1) The business requirement (or question) q input by the user is: Please output the number of inactive broadband users last month by province in 2023; The first historical reference example Aw1 is retrieved through the vector database. For example, in the vector database, {input: "user question (such as business requirement)"}; {output: "historical requirement (or historical question)", "historical (result) table name"}, and the execution result is shown in Figure 3 , then T1 = {TableA, TableB}, p1 = 2;

[0073] 2) Fill the content in Aw1 into the reference example part (including historical questions and result tables) of the first prompt template shown in Figure 4 , and fill the table information (including table name and table description) of the result table and the user question q into the SQL table information and question part of the first prompt template, then p1 first prompts can be obtained, such as Figure 5a and Figure 5b shown;

[0074] 3) After inputting the first prompts shown in Figure 5a and Figure 5b into the semantic large model, Aw2 is obtained as shown in Figure 6 , then T2 = {TableA, TableC};

[0075] 4) Take the union of T1 and T2 to obtain the first reply content Tr = {TableA, TableB, TableC}.

[0076] It should be noted that the prompt is the input of the large model, and different prompts can obtain different text outputs. The role of the above prompts is to let the large model find the table information corresponding to the current requirement in the vector database.

[0077] In step S206 of the above data processing method, generate prompts corresponding to each historical reference example according to the prompt template, and use the large model to generate the output results of the prompts, including: in the case where the business requirement is field query, determine the second historical reference example corresponding to the business requirement, where the second historical reference example includes historical requirements, historical table names, and historical field names, and the historical table name is the table name corresponding to the table set selected from the first output result generated by the large model when performing table query on the business requirement; generate second prompts corresponding to each second historical reference example according to the second prompt template, and use the large model to generate the second output results of the second prompts, where the second output results include the reasons for selecting fields and the set of selected fields.

[0078] In some embodiments of the present application, the system first identifies that the business requirement proposed by the user contains a field query. Based on the identified field query requirement, the system filters out a second set of historical reference examples from the vector database. Different from table queries, the second set of historical reference examples contains, in addition to the historical requirement description, the historical table names and historical field names involved in the history. It should be noted that the historical table names are the table names covered by the set of tables selected from the first output result generated by the large model in the previous table query scenario. Therefore, it can be ensured that the field query is carried out within the scope of relevant tables. The system generates a series of second prompts based on the second prompt template and the selected second set of historical reference examples. The generated second prompts are submitted to the large model for prediction. The large model generates a second output result based on the second prompt information and the knowledge it has learned from a large amount of data. The second output result includes field recommendations for the current business requirement and the reasons for recommending each field.

[0079] Through this process, when the user proposes a field query requirement, the system can, through the combination of historical reference examples and large model prediction, intelligently recommend the fields involved and the reasons for their selection, so as to provide accurate query suggestions for the user.

[0080] In step S208 of the above data processing method, according to the historical reference examples and the output result, the reply content corresponding to the business requirement is determined, including: in the case where the output result is the second output result, obtaining the historical field set composed of the historical field names in the second set of historical reference examples; merging the historical field set and the field set selected in the second output result to obtain a second union; after de-duplicating the second union, obtaining the second reply content corresponding to the business requirement.

[0081] In some embodiments of the present application, the system extracts all the involved field names from the second set of historical reference examples to obtain a historical field set, and merges the historical field set with the field set recommended (or called the selected field set) in the second output result generated by the large model to obtain a second union. To ensure the accuracy of the reply content and avoid redundancy, the system will perform de-duplication processing on the second union, removing duplicate field names and only retaining the uniquely occurring fields to obtain the second reply content.

[0082] The process of field query is described in detail below in combination with a specific process, including the following steps:

[0083] 1) According to the user's business requirement q, within the scope of the first reply content Tr, p2 second historical reference examples Aw3 are retrieved from the vector database through the vectorization model. Aw3 includes {historical requirement, historical table name, historical field name}. Let {historical table name, historical field name} in Aw3 be the historical field set C1;

[0084] 2) Use the second prompt template to generate p2 second prompts by combining each second historical reference example;

[0085] 3) Call the semantic large model one by one for the p2 second prompts, and each second prompt obtains n2 output results Aw4 (n2≥1), where Aw4 includes {reasons for selecting fields, selected field set}. After the p2 second prompts are executed, p2*n2 results Aw4 are obtained, that is, the second output result is obtained. Let the selected field set in Aw4 be C2;

[0086] 4) Take the union of C1 and C2 as the final second reply content Cr.

[0087] In some embodiments of the present application, p2≥p1.

[0088] The above process can be illustrated by the following example. For example:

[0089] 1) According to the table query processing of the database, Tr = {TableA, TableB, TableC} is obtained. The second historical reference example Aw3 is obtained through vector retrieval, {input: "user question (such as business requirements)"}; {output: "historical requirements (or historical questions)", "historical output results"}, where the historical output results include historical table names and historical field names, and the execution results are as Figure 7 shown. Then C1 = {"TableA": ["month", "active", "user_id"], "TableB": ["month", "active", "user_id", "city"]}, and p2 = 2;

[0090] 2) Fill the content in Aw3 into the reference example part (including historical questions and output results) of the second prompt template shown in Figure 8 . Fill the table information (including table name, field name, field description) in Aw3 and the user question q into the SQL field information and question part of the second prompt template, then p2 second prompts can be obtained, such as Figure 9a and Figure 9b shown;

[0091] 3) After inputting the second prompts shown in Figure 9a and Figure 9b into the semantic large model, Aw4 is obtained as shown in Figure 10 . Then C2 = {"TableA": ["year", "active", "user_id", "province", "user_type"], "TableB": ["month", "active", "user_id", "city", "user_type"]};

[0092] 4) Take the union of C1 and C2, then Cr = {"TableA": ["year", "active", "user_id", "province", "user_type"], "TableB": ["month", "active", "user_id", "city", "user_type"]}.

[0093] Combining the above processes, the final content input to NL2SQL includes the first response content Tr and the second response content Cr to better generate SQL.

[0094] In the above data processing method, the vector database where the historical reference samples are located is determined in the following ways, including two methods: Method 1: Obtain the historical retrieval records stored locally; verify the historical retrieval records to obtain historical question pairs, where the historical question pairs include historical questions, historical table information, and historical field information; use a vectorization model to vectorize the historical questions to obtain historical question vectors; store the historical question vectors, historical questions, historical table information, and historical field information in the vector database; or, Method 2: Obtain the historical query results corresponding to the historical questions through an SQL statement generation framework, where the historical query results include historical table information and historical field information; in the case where the historical query results are normal data, for example, if the user confirms that the historical query results are available, then consider the historical query results as normal data, and determine the historical query results and historical questions as historical question pairs; use a vectorization model to vectorize the historical questions to obtain historical question vectors; store the historical question vectors, historical questions, and historical query results in the vector database.

[0095] When forming the above historical question pairs, the following steps may be included:

[0096] 1) Input the question and the SQL statement, and parse out the table and field information in the SQL through the SQL anti-parsing module. The input example is as Figure 11 shown, including the result of the question and the SQL statement; the output example is as Figure 12 shown, including the reason, the table set, and the column set (i.e., the field set);

[0097] 2) After vectorizing the question, store it in the vector database together with the output example, as Figure 13 shown.

[0098] The data processing method provided by the embodiments of the present application, in the case of processing multiple tables or a large number of fields, through a method of layer-by-layer screening by combining a prompt with vectorized retrieval, narrows down the range of tables and fields that may be needed, thereby improving the retrieval efficiency. In addition, by looking for historical approximate references from the perspective of semantic relevance, the accuracy of the final screening is further improved. This data processing method only needs to enable the framework to process new information through the method of table / field annotations, and the large model further determines whether the required tables / fields according to the annotations, avoiding the problem that keywords cannot be accurately matched and also solving the problems of abbreviations and shorthands.

[0099] Figure 14 is a structural diagram of a data processing device according to an embodiment of the present application, as Figure 14 shown, the device includes:

[0100] An acquisition module 40, configured to acquire the business requirements input by the target object;

[0101] A first determination module 42, configured to determine a historical reference example corresponding to the business requirements, where the historical reference example includes historical requirements and historical query content;

[0102] A generation module 44, configured to generate a prompt corresponding to each historical reference example according to a prompt template, and generate an output result of the prompt by using a large model;

[0103] A second determination module 46, configured to determine a reply content corresponding to the business requirements according to the historical reference example and the output result.

[0104] Through the acquisition module 40, the first determination module 42, the generation module 44, and the second determination module 46 in the above data processing device, the purpose of accurately identifying and screening relevant tables and fields in the database is achieved, thereby realizing the technical effect of improving the accuracy of NL2SQL conversion, and further solving the technical problem that it is difficult for the large model to accurately extract relevant information from a complex database when generating an SQL statement, resulting in a low accuracy of the generated SQL statement.

[0105] In the above data processing device, when the business requirement is a table query, the historical query content includes the historical table name; when the business requirement is a field query, the historical query content includes the historical table name and the historical field name.

[0106] In the generation module of the above data processing device, the generation module is further configured to, when the service requirement is a table query, determine a first historical reference example corresponding to the service requirement, where the first historical reference example includes a historical requirement and a historical table name; generate a first prompt corresponding to each first historical reference example according to the first prompt template, and use a large model to generate a first output result of the first prompt, where the first output result includes the reason for selecting the table and the selected table set.

[0107] In the second determination module of the above data processing device, the second determination module is further configured to, when the output result is the first output result, obtain a historical table set composed of the historical table names in the first historical reference example; merge the historical table set and the selected table set in the first output result to obtain a first union; after performing deduplication processing on the first union, obtain a first reply content corresponding to the service requirement.

[0108] In the generation module of the above data processing device, the generation module is further configured to, when the service requirement is a field query, determine a second historical reference example corresponding to the service requirement, where the second historical reference example includes a historical requirement, a historical table name, and a historical field name, and the historical table name is the table name corresponding to the selected table set in the first output result generated by the large model when performing a table query on the service requirement; generate a second prompt corresponding to each second historical reference example according to the second prompt template, and use a large model to generate a second output result of the second prompt, where the second output result includes the reason for selecting the field and the selected field set.

[0109] In the second determination module of the above data processing device, the second determination module is further configured to, when the output result is the second output result, obtain a historical field set composed of the historical field names in the second historical reference example; merge the historical field set and the selected field set in the second output result to obtain a second union; after performing deduplication processing on the second union, obtain a second reply content corresponding to the service requirement.

[0110] In the first determination module of the above data processing device, the first determination module is further configured to determine the vector database where the historical reference example is located. Specifically, the vector database where the historical reference example is located is determined by the following method: Obtain the historical retrieval records stored locally; Verify the historical retrieval records to obtain historical question pairs, where the historical question pairs include historical questions, historical table information, and historical field information; Use a vectorization model to vectorize the historical questions to obtain historical question vectors; Store the historical question vectors, historical questions, historical table information, and historical field information in the vector database; Or, obtain the historical query result corresponding to the historical question through an SQL statement generation framework, where the historical query result includes historical table information and historical field information; In the case where the historical query result is normal data, determine the historical query result and the historical question as a historical question pair; Use a vectorization model to vectorize the historical questions to obtain historical question vectors; Store the historical question vectors, historical questions, and historical query results in the vector database.

[0111] It should be noted that Figure 14 the data processing device shown is used to execute Figure 2 the data processing method shown. Therefore, the relevant explanations in the above data processing method also apply to this data processing device, and will not be elaborated here.

[0112] An embodiment of this application also provides an electronic device, which includes a memory and a processor. Among them, the memory is used to store program instructions; The processor is connected to the memory and is used to execute program instructions to implement the following functions: Obtain the business requirements input by the target object; Determine the historical reference example corresponding to the business requirements, where the historical reference example includes historical requirements and historical query content; Generate a prompt corresponding to each historical reference example according to the prompt template, and use a large model to generate the output result of the prompt; Determine the reply content corresponding to the business requirements based on the historical reference example and the output result.

[0113] It should be noted that the above electronic device is used to execute Figure 2 the data processing method shown. Therefore, the relevant explanations in the above data processing method also apply to this electronic device, and will not be elaborated here.

[0114] An embodiment of the present application also provides a non - volatile storage medium, which includes a stored computer program. Wherein, the device where the non - volatile storage medium is located executes the following data processing method by running the computer program: obtaining the service requirements input by the target object; determining the historical reference examples corresponding to the service requirements, where the historical reference examples include historical requirements and historical query contents; generating prompts corresponding to each historical reference example according to the prompt template, and using a large - model to generate the output results of the prompts; determining the response content corresponding to the service requirements based on the historical reference examples and the output results.

[0115] It should be noted that the above - mentioned non - volatile storage medium is used to execute Figure 2 the data processing method shown. Therefore, the relevant explanations in the above - mentioned data processing method also apply to this non - volatile storage medium, and will not be elaborated here.

[0116] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the data processing method in each embodiment of the present application when executed by a processor.

[0117] An embodiment of the present application also provides a computer program, which implements the steps of the data processing method in each embodiment of the present application when executed by a processor.

[0118] The serial numbers of the above - mentioned embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments.

[0119] In the above - mentioned embodiments of the present application, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0120] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be electrical or other forms.

[0121] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0123] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0124] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A data processing method, characterized in that, Including: Obtain the business requirements input by the target object; Determine the historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query content, and the historical query content includes historical table names and / or historical field names; Generate prompts corresponding to each historical reference example according to the prompt template, and use a large model to generate the output results of the prompts; Determine the response content corresponding to the business requirements based on the historical reference examples and the output results.

2. The method according to claim 1, wherein The method further includes: When the business requirement is a table query, the historical query content includes historical table names; When the business requirement is a field query, the historical query content includes historical table names and historical field names.

3. The method according to claim 1, characterized in that, Generating prompts corresponding to each historical reference example according to the prompt template and using a large model to generate the output results of the prompts includes: When the business requirement is a table query, determine the first historical reference example corresponding to the business requirement, where the first historical reference example includes historical requirements and historical table names; Generate first prompts corresponding to each first historical reference example according to the first prompt template, and use the large model to generate the first output results of the first prompts, where the first output results include the reasons for selecting tables and the set of selected tables.

4. The method according to claim 3, characterized in that, Determining the response content corresponding to the business requirements based on the historical reference examples and the output results includes: When the output result is the first output result, obtain the historical table set composed of the historical table names in the first historical reference example; Merge the historical table set and the set of selected tables in the first output result to obtain a first union; after de-duplicating the first union, obtain the first response content corresponding to the business requirements.

5. The method according to claim 3, wherein Generating prompts corresponding to each historical reference example according to the prompt template and using a large model to generate the output results of the prompts includes: When the business requirement is a field query, determine the second historical reference example corresponding to the business requirement, where the second historical reference example includes historical requirements, historical table names, and historical field names, and the historical table name is the table name corresponding to the set of selected tables in the first output result generated by the large model when performing a table query on the business requirement; Generate second prompts corresponding to each second historical reference example according to the second prompt template, and use the large model to generate the second output results of the second prompts, where the second output results include the reasons for selecting fields and the set of selected fields.

6. The method according to claim 5, characterized in that Determining the response content corresponding to the business requirements based on the historical reference examples and the output results includes: When the output result is the second output result, obtain the historical field set composed of the historical field names in the second historical reference example; Merge the historical field set and the set of selected fields in the second output result to obtain a second union; After de-duplicating the second union, obtain the second response content corresponding to the business requirements.

7. The method according to claim 1, characterized in that The vector database where the historical reference examples are located is determined as follows: Obtain the historical retrieval records stored locally; Verify the historical retrieval records to obtain historical question pairs, where the historical question pairs include historical questions, historical table information, and historical field information; Use a vectorization model to vectorize the historical questions to obtain historical question vectors; Store the historical question vectors, the historical questions, the historical table information, and the historical field information in the vector database; Or, Obtain the historical query results corresponding to the historical questions through an SQL statement generation framework, where the historical query results include the historical table information and the historical field information; When the historical query results are normal data, determine the historical query results and the historical questions as the historical question pairs; Use a vectorization model to vectorize the historical questions to obtain historical question vectors; Store the historical question vectors, the historical questions, and the historical query results in the vector database.

8. A data processing device, characterized in that, It includes: An acquisition module for acquiring the business requirements input by the target object; A first determination module for determining historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents; A generation module for generating prompts corresponding to each historical reference example according to a prompt template, and using a large model to generate the output results of the prompts; A second determination module for determining the reply content corresponding to the business requirements according to the historical reference examples and the output results.

9. An electronic device, characterized in that, It includes: A memory for storing program instructions; A processor, connected to the memory, for executing program instructions to implement the following functions: acquiring the business requirements input by the target object; determining historical reference examples corresponding to the business requirements, where the historical reference examples include historical requirements and historical query contents; generating prompts corresponding to each historical reference example according to a prompt template, and using a large model to generate the output results of the prompts; Determining the reply content corresponding to the business requirements according to the historical reference examples and the output results.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, where the device where the non-volatile storage medium is located executes the data processing method according to any one of claims 1 to 7 by running the computer program.

Images