Data processing method and device based on large model, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device based on a large model, equipment and a medium. In the embodiment of the invention, the electronic equipment performs multi-task mixed training on a large amount of sample data and the corresponding natural language description to obtain the large model, so that the large model is trained to complete multiple types of tasks. In the reasoning stage, the electronic equipment can perform text classification on the received target text so as to determine the to-be-executed target task type, and meanwhile, based on the dynamic example recall strategy, the cue word of the target task type is determined, and the cue word and the target text are input into the large model, so that the understanding generation capability of the large model is improved, and the user experience is improved. According to the method, the technical threshold of data access is greatly reduced, the user experience is improved, the interaction friendliness is enhanced, and the like, so that a large model is comprehensively applied to scenes in the field of SQL (Structured Query Language).

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium based on a large model. Background Art

[0002] With the development of artificial intelligence technology, the use scenarios of big models are becoming more and more extensive. However, in the current use scenarios, big models are only used for Text2SQL query applications. For example, when the big model receives the question text "Query what is the GDP of Qingdao City", the big model generates SQL, and the electronic device gives the user a reply based on the SQL. This results in the big model only playing a partial role and failing to give full play to the other capabilities of the big model in SQL application, that is, the application efficiency of the big model is low. Summary of the invention

[0003] The present application provides a data processing method, device, equipment and medium based on a big model, which is used to solve the problem that the big model only plays a partial role and fails to give full play to other capabilities of the big model in SQL application, that is, the application efficiency of the big model is low.

[0004] In a first aspect, an embodiment of the present application provides a data processing method based on a large model, the method comprising:

[0005] Receive a target text input by a user, and determine a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display;

[0006] Obtaining a pre-configured data processing prompt text corresponding to the target intent, and inputting the prompt text and the target text into a large model; wherein the large model is obtained by mixed training based on sample data of multiple tasks;

[0007] The large model is scheduled to execute the task of the target intention, and the processing result output by the large model is obtained.

[0008] In a second aspect, an embodiment of the present application further provides a data processing device based on a large model, the device comprising:

[0009] An intention recognition module, used to receive a target text input by a user and determine a target intention of the target text; wherein the target intention is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display;

[0010] A processing module is used to obtain a pre-configured data processing prompt text corresponding to the target intent, and input the prompt text and the target text into a large model; wherein the large model is obtained by mixed training based on sample data of multiple tasks; the large model is scheduled to execute the task of the target intent, and obtain the processing result output by the large model.

[0011] In a third aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned large model-based data processing methods when executing a computer program stored in the memory.

[0012] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the large model-based data processing methods described above.

[0013] In the embodiment of the present application, the electronic device obtains a large model by performing multi-task mixed training on a large amount of sample data and its corresponding natural language description, thereby realizing the training of the large model to complete multiple types of tasks. In the reasoning stage, the electronic device can perform text classification on the received target text to determine the type of target task to be performed, and at the same time, based on the dynamic example recall strategy, determine the prompt word of the target task type, and input the prompt word and the target text into the large model, thereby improving the understanding and generation ability of the large model. The present application greatly reduces the technical threshold for data access, improves user experience, enhances interactive friendliness, and has significant advantages, enabling the comprehensive application of large models in SQL field scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A schematic diagram of a data processing process based on a large model provided in an embodiment of the present application;

[0016] Figure 2 A functional schematic diagram of an electronic device provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a fine-tuning process for an embodiment of the present application;

[0018] Figure 4 A schematic diagram of a fine-tuning process for multiple purposes provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of fine-tuning a large model provided in an embodiment of the present application;

[0020] Figure 6 A data processing flow chart provided for an embodiment of the present application;

[0021] Figure 7 A data processing flow chart of the Text2SQL intent provided in an embodiment of the present application;

[0022] Figure 8 A schematic diagram of the data processing process of the multi-intention text provided in the embodiment of the present application;

[0023] Fig. 9 A schematic diagram of the hybrid search process provided in the embodiment of the present application;

[0024] Fig.10 A schematic diagram of the structure of a data processing device based on a large model provided in an embodiment of the present application;

[0025] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0027] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of big models in SQL field scenarios, the embodiments of the present application provide a data processing method, device, equipment and medium based on a big model.

[0028] In an embodiment of the present application, an electronic device receives a target text input by a user, and determines a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display; obtains a data processing prompt text corresponding to a pre-configured target intent, and inputs the prompt text and the target text into a large model; wherein the large model is obtained by mixed training based on sample data of multiple tasks; schedules the large model to execute tasks of the target intent, and obtains the processing results output by the large model.

[0029] Figure 1A schematic diagram of a data processing process based on a large model provided in an embodiment of the present application, the process includes:

[0030] S101: Receive a target text input by a user, and determine a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display.

[0031] A large model-based data processing method provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.

[0032] In the embodiments of the present application, the electronic device performs hybrid training on the big model based on sample data of multiple tasks, so that the big model can complete multiple tasks, greatly reducing the technical threshold for data access, improving user experience, enhancing interactive friendliness, etc. It has significant advantages, and enables the big model to be fully applied in SQL field scenarios.

[0033] Based on this, the user can input the target text to be processed into the electronic device by inputting text. After the electronic device receives the target text, the electronic device needs to perform what tasks for the large language model of the target text, thereby dynamically constructing the prompt text and scheduling the large model.

[0034] Specifically, the electronic device performs intent recognition on the target text and determines the target intent corresponding to the target text, where one intent corresponds to one task. The electronic device can use a variety of methods to recognize the intent of the target text, including but not limited to: inputting the target text into a trained intent recognition model, scheduling the intent recognition model to perform intent recognition on the target text, and determining the target intent of the target text; or, performing keyword detection on the target text, and determining the intent corresponding to the target keyword as the target intent of the target text based on the detected target keyword and the correspondence between the pre-configured keyword and the intent; or, inputting the target text into a large language model, scheduling the large language model to determine and output the target intent of the target text. The intent recognition model can also be a classification model, etc., which is not limited here.

[0035] In an embodiment of the present application, the target intent of the target text determined by the electronic device is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display.

[0036] Among them, library and table management includes but is not limited to database management and table management. Database management includes but is not limited to creating a new database table, deleting a database table, and modifying the name of a data table. For example, if the target text entered by the user is "Create ai_test database", the electronic device determines that the intention of the target text is library and table management. Table management includes but is not limited to creating table fields, modifying table fields, and deleting table fields. Among them, the creation of table fields is changed from manual assignment of types, adding field attribute introductions, and designing table field names to automatic creation. For example, the target text entered by the user is "Create a store sales data table with fields including year, month, day, full date, store name, first-level region, second-level region, first-level channel, store daily operating sales, store daily fixed costs, beginning inventory of the day, and ending inventory of the day, and generate appropriate indexes for table fields", then the electronic device determines that the target intention of the target text is library and table management.

[0037] Text2SQL generates an SQL statement based on the target text input by the user. For example, if the target text is "how many people enter the store every day", the electronic device determines that the target intent is Text2SQL.

[0038] The target text input by the user in SQL2Text is a SQL statement. The electronic device needs to convert the SQL statement into a text in a natural language. For example, if the target text is "SELECT*FROM user WHERE age>30", the electronic device determines that the target intent is SQL2Text.

[0039] In a possible implementation, if the target text is "SELECT*FROM user WHERE age>30", the processing result determined by the electronic device includes but is not limited to statement interpretation and statement optimization.

[0040] Among them, the statement explanation can be:

[0041] The meaning of this SQL statement is to extract all user records whose age is greater than 30 from the users table.

[0042] This SQL statement is a typical query statement used to retrieve data from a database with specific conditions. Specifically, it selects data from all columns (indicated by the asterisk *) from the users table, but only includes those records whose age (age is greater than 30. This query can be used for a variety of purposes, for example, if the user wants to view the details of all adult users, or to perform some kind of data analysis based on age.

[0043] The components of this statement explain its function:

[0044] SELECT* means selecting data from all columns.

[0045] FROM users specifies which table to extract data from, here is the users table

[0046] WHERE age>30 is a conditional clause that limits the query results to only those records whose age is greater than 30.

[0047] The statement optimization can be: SELECT id, name, email, age FROM user WHERE age>30.

[0048] Data generation can be used to generate training data for large models or some simulated data for model training. It includes but is not limited to generating data and modifying the existing database table structure, such as adding new columns. For example, if the target text is "Help me generate 15 pieces of data information that are close to the real thing. The first-level region includes the southwest, northwest, etc. The first-level southwest region includes second-level cities such as Chengdu, Guiyang, and Chongqing, and the first-level northwest region includes second-level cities such as Lanzhou and Xi'an...", then the electronic device determines that the target intent is data generation; if the target text is "Add a column for the number of visitors to the store, the column name is English, and randomly fill in the existing data", then the electronic device determines that the target intent is data generation.

[0049] Table field optimization is to convert non-standard table fields into table fields by using a large model to deeply understand the field names and field values ​​of the source database system and convert them into the best standard syntax of the target database system. For example, if the target text is "help me optimize the ai_test table fields", the electronic device determines that the target intent is table field optimization.

[0050] In the embodiment of the present application, the content that can be optimized in the table field optimization includes but is not limited to: changing pinyin to professional English, changing professional English according to annotations, changing wrong words to correct words and type optimization, etc., all of which are converted into vocabulary that is easier to understand for the large model.

[0051] Chart display supports chart display of numerical data in the database, and can support display in the form of bar charts, line charts, pie charts, rose charts, bar charts, etc. The data distribution in the database can be displayed at a glance. For example, if the target text is "the horizontal axis is the region, the vertical axis is 100 million yuan, and the sales volume of the product is displayed in a bar chart", the electronic device determines the target intention as chart display; if the target text is "display with a bar chart", the electronic device determines the target intention as chart display.

[0052] S102: Obtain the pre-configured data processing prompt text corresponding to the target intent, and input the prompt text and the target text into the big model; wherein the big model is obtained by mixed training based on sample data of multiple tasks; schedule the big model to execute the task of the target intent, and obtain the processing result output by the big model.

[0053] In the embodiment of the present application, when training the large model, different data processing prompt texts are used for different intentions. The large model can be a large language model, such as GPT-3, Wenyan Yixin, etc., or a multimodal large model, such as GPT-4, etc., which is not limited here.

[0054] Based on this, in the embodiment of the present application, after the electronic device determines the target intent of the target text, the electronic device also obtains the data processing prompt text corresponding to the pre-configured target intent, inputs the prompt text and the target text into the large model, so that the large model processes the target text according to the data processing prompt text. The large model is obtained by mixed training based on sample data of multiple tasks.

[0055] In an embodiment of the present application, the electronic device schedules the large model to execute the task corresponding to the target intention through data processing prompt text, so that the large model determines and outputs the processing result for the target text.

[0056] Each intent corresponds to a data prompt text, which includes but is not limited to task instructions, task constraints and business constraints. The task instruction is used to prompt the big model which task corresponding to the intent currently needs to be executed. For example, the task instruction corresponding to the SQL2Text intent is "You are now a database expert. Your current task is to generate corresponding explanations and optimize SQL based on the SQL input by the user"; task constraints and business constraints are used to constrain the processing of the big model. For example, the task constraints and business constraints corresponding to the SQL2Text intent are "##**Always remember that the output result must be an SQL statement**; ##Try not to use * when querying all, and use the field name in the database; ##SQL must give a corresponding explanation, and if SQL cannot be optimized, it will be output as is".

[0057] Figure 2 A functional schematic diagram of an electronic device provided in an embodiment of the present application, such as the Figure 2 As shown, the electronic device performs text fusion and text classification on user questions, and can realize tasks such as library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display.

[0058] In the embodiment of the present application, the electronic device obtains a large model by performing multi-task mixed training on a large amount of sample data and its corresponding natural language description, thereby realizing the training of the large model to complete multiple types of tasks. In the reasoning stage, the electronic device can perform text classification on the received target text to determine the type of target task to be performed, and at the same time, based on the dynamic example recall strategy, determine the prompt word of the target task type, and input the prompt word and the target text into the large model, thereby improving the understanding and generation ability of the large model. The present application greatly reduces the technical threshold for data access, improves user experience, enhances interactive friendliness, and has significant advantages, enabling the comprehensive application of large models in SQL field scenarios.

[0059] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of big models in SQL field scenarios, based on the above embodiments, in the embodiments of this application, the training process of the big model includes:

[0060] For each intent, obtain the pre-configured data processing prompt text corresponding to the intent, wherein the data processing prompt text includes the task instructions, task constraints and business constraints corresponding to the target intent; input the data processing prompt text, pre-configured sample data and corresponding sample processing results into the big model; fine-tune the big model according to the lora fine-tuning algorithm to determine the parameters used by the big model when executing the task corresponding to the intent.

[0061] In order to make the large model more applicable to multiple tasks, in the embodiment of the present application, the electronic device will use sample data of multiple tasks to train the large model. Specifically, the embodiment of the present application adopts the lora method for training, and there is basically no innovation in the lora method itself, but in order to improve the performance of the large model, the embodiment of the present application introduces a dedicated LoRA module for each task to improve the performance of the large model in multi-task prediction tasks.

[0062] Specifically, for each intent, the electronic device obtains the pre-configured data processing prompt text corresponding to the intent, wherein the data processing prompt text includes the task instructions, task constraints and business constraints corresponding to the target intent. The electronic device inputs the data processing prompt text, the pre-configured sample data and the corresponding sample processing results into the large model, and fine-tunes the large model according to the lora fine-tuning algorithm to determine the parameters used by the large model when executing the task corresponding to the intent.

[0063] Figure 3 A schematic diagram of a fine-tuning process for an embodiment of the present application is provided, such as the Figure 3 As shown, when training the large model, the parameters of the large model need to be updated, which can be expressed as W0 +ΔW, where W 0 is the initialization parameter, and ΔW is the parameter that needs to be updated. For the lora fine-tuning algorithm, only ΔW needs to be fine-tuned.

[0064] Specifically, assume that the parameter W of the large model is 0 ∈R d×k , his update can be expressed as:

[0065] W 0 +ΔW=W 0 +BA,B∈R d×r ,A∈R r×k

[0066] Among them, W 0 is fixed, only A and B are training parameters, and the rank r<<min(d,k).

[0067] Figure 4 A schematic diagram of a fine-tuning process for multiple purposes provided in an embodiment of the present application, such as the Figure 4 As shown, low-rank matrices are used as parameters of the large model for each intent to learn personalized information for each intent. It can be expressed as follows, where t represents the tth task:

[0068] y=W+ΔW t =W+BA

[0069] W represents the pre-training parameters, A and B are the training parameters, t represents the task corresponding to the tth intent, and y represents all the parameters of the large model.

[0070] Figure 5 The schematic diagram of fine-tuning the large model provided in the embodiment of the present application is Figure 5 As shown, the existing base model has a low accuracy problem in the current Text2SQL and SQL2Text tasks. To address this problem, the present application embodiment carries out large model fine-tuning and prompt word engineering, including calling the base model for single-task training and dynamic example recall, so that the large model can be used for library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display.

[0071] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness and other aspects, and enable the comprehensive application of large models in SQL field scenarios, on the basis of the above embodiments, in the embodiments of the present application, if the intention is Text2SQL or SQL2Text, the data processing prompt text also includes database table structure information.

[0072] In the embodiment of the present application, for Text2SQL or SQL2Text, in order to enable the large language model to better process the target text, the data processing prompt text corresponding to Text2SQL or SQL2Text also includes database table structure information.

[0073] The database table structure information includes, but is not limited to, the name of the database table, the name of the field, and the value corresponding to the field.

[0074] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of large models in SQL field scenarios, based on the above embodiments, in the embodiment of the present application, the process of determining the database table structure information includes:

[0075] Determine the target feature vector corresponding to the target text according to a preset embedding representation algorithm;

[0076] Obtaining each first feature vector corresponding to the saved summary description of each database table, and determining a first preset number of target database tables that are most similar to the target text according to each first feature vector and the target feature vector;

[0077] Obtaining each second feature vector corresponding to each field description of the target database table that is saved, and determining a second preset number of target fields that are most similar to the target text according to the second feature vector and the target feature vector;

[0078] The table names of the first preset number of target database tables and the field information of the second preset number of target fields are determined as the database table structure information.

[0079] In an embodiment of the present application, when the electronic device determines the database table structure information, the electronic device may determine it based on table recall, field recall, enumeration value recall, and the like.

[0080] Specifically, the electronic device preliminarily summarizes each database table in the database, describes the purpose of each database table, and explains which fields each database table can support user inquiries. At the same time, the electronic device describes the fields of each database table. The electronic device uses vector models such as BGE and BCE to embed the generated summary description, field description, and enumeration value, and obtains and saves the first feature vector corresponding to each summary description, the second feature vector corresponding to each field description, and the third feature vector corresponding to each enumeration value.

[0081] Based on this, when the electronic device determines the data table structure information corresponding to the target text, the electronic device determines the target feature vector corresponding to the target text according to the preset embedding representation algorithm; obtains each first feature vector corresponding to the summary description of each stored database table, and determines the first preset number of target database tables that are most similar to the target text according to each first feature vector and the target feature vector. The electronic device obtains each second feature vector corresponding to each field description of the stored target database table, and determines the second preset number of target fields that are most similar to the target text according to the second feature vector and the target feature vector. The electronic device determines the table names of the first preset number of target database tables and the field information of the second preset number of target fields as the database table structure information.

[0082] The field information includes, but is not limited to, the field name, field description, and the value corresponding to the field.

[0083] In addition, in an embodiment of the present application, if the target field is an enumeration type field, the electronic device may also obtain each enumeration value corresponding to the target field and the corresponding third feature vector stored therein, and determine the second preset number of target enumeration values ​​that are most similar to the target text according to the third feature vector and the target feature vector. The electronic device determines the table names of the first preset number of target database tables, the field names of the second preset number of target fields, and the third preset number of target enumeration values ​​as database table structure information.

[0084] In the embodiment of the present application, the first preset number, the second preset number and the third preset number can be configured according to user needs, but the value cannot exceed the maximum value that can be selected. That is, the first preset number shall not exceed the total number of database tables included in the database, the second preset number shall not exceed the total number of fields included in the target database table, and the third preset number shall not exceed the total number of enumeration values ​​included in the target field.

[0085] In one possible manner, the electronic device may use a cosine similarity algorithm to perform vector similarity calculations, such as: Res=cosine(Embedding(target problem), Embedding(summary description / field description / enumeration value)).

[0086] It should be noted that, in the embodiment of the present application, when determining the target field, the second preset number of target fields may be screened out from each target database table, or the second preset number of target fields may be screened out. Generally, the second preset number of target fields are screened out from each target database table.

[0087] Specifically, after completing the similarity calculation, the electronic device obtains the Top-K target database tables above the threshold, and obtains the Top-N target fields in each target database table to form the database structure information. The purpose is to enhance the generalization of the large model after training, improve the large model's ability to distinguish similar fields, and at the same time save the length of the large model input token and reduce the impact of redundant data on the large model input.

[0088] On the basis of the above-mentioned embodiment, in the embodiment of the present application, the electronic device can input the data processing prompt text and the target question into the large model for processing.

[0089] Figure 6 The data processing flow chart provided in the embodiment of the present application is as follows: Figure 6 The process includes: the electronic device performs text classification on the user question (target question) to determine that the intention of the user question is Text2SQL; the electronic device performs field recall, table recall and enumeration value recall to determine the database table structure information; the electronic device inputs the database table structure information, business constraints and instructions into the big model.

[0090] In a possible implementation, the input of the large model may be:

[0091]

[0092] Table 1

[0093] Table 1 is a large model input of the Text2SQL intent provided in an embodiment of the present application. As shown in Table 1, the input includes task instructions, database table structure information, task constraints + business constraints, and target text in natural language format.

[0094] In another possible implementation, the input of the large model may be:

[0095]

[0096]

[0097] Table 2

[0098] Table 2 is a large model input of the SQL2Text intent provided in an embodiment of the present application. As shown in Table 2, the input includes task instructions, database table structure information, task constraints + business constraints and target text in SQL format.

[0099] It is worth noting that, except for Text2SQL and SQL2Text intents, when processing the target text of other intents, there is no need to recall the database table structure information. You only need to input the data processing prompt text containing task instructions + task constraints + business constraints and the target text into the large language model.

[0100] Moreover, in other intents, different data processing prompt texts are constructed according to different intents for large model training, and a set of data processing prompt texts is designed for each intent for training.

[0101] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of large models in SQL field scenarios, based on the above embodiments, in the embodiment of the present application, if the target intent is Text2SQL, the processing result of obtaining the output of the large model includes:

[0102] Obtain the target SQL statement output by the large model;

[0103] The method further comprises:

[0104] Execute the target SQL statement;

[0105] If the execution fails, the pre-configured SQL repair prompt text and the target SQL statement are input into the large model so that the large model repairs the target SQL statement;

[0106] Obtain the repaired target SQL statement output by the large model.

[0107] In the embodiment of the present application, after the large language model outputs the target SQL statement, the electronic device obtains the target SQL statement and sends the target SQL statement into the data for execution. If the execution is successful, there is no syntax error and it will be output. If the execution recognition indicates that there is a syntax error, the electronic device constructs a new prompt and allows the large model to perform SQL error correction.

[0108] Specifically, the electronic device obtains the target SQL statement output by the large model and executes the target SQL statement. If the execution fails, the electronic device inputs the pre-configured SQL repair prompt text and the target SQL statement into the large model, so that the large model repairs the target SQL statement and obtains the repaired target SQL statement output by the large model.

[0109] The SQL repair prompt text includes but is not limited to task instructions, database table structure information, task constraints + business constraints and examples.

[0110] In a possible implementation, when the electronic device calls the large language model to perform SQL repair, the input of the large model may be:

[0111]

[0112]

[0113] Table 3

[0114] Table 3 is a large model input for SQL repair provided in an embodiment of the present application. As shown in Table 3, the input includes task instructions, database table structure information, task constraints + business constraints, examples and target SQL statements.

[0115] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of large models in SQL field scenarios, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0116] Execute the repaired target SQL statement;

[0117] If the execution fails, the response information of processing failure will be output.

[0118] In the embodiment of the present application, after the electronic device calls the large model to repair the target SQL statement, the electronic device will execute the repaired target SQL statement again. If the execution result of the repaired target SQL statement is still execution failure, the electronic device outputs a response message of processing failure.

[0119] Specifically, after correcting the syntax of the target SQL statement, the electronic device schedules the SQL executor to run the code of the repaired target SQL statement. If the execution is successful, the repaired target SQL statement is output as a reference. If errors continue to occur during SQL execution, it is discarded and no longer used, and a response message is sent to the user that the SQL cannot be generated.

[0120] Figure 7 The data processing flow chart of the Text2SQL intent provided in the embodiment of the present application is as follows: Figure 7 As shown, the electronic device calls a large language model (LLM) to perform Text2SQL on the target text to obtain a target SQL statement; the electronic device calls an SQL executor to execute the target SQL statement; the electronic device determines whether the target SQL statement is executed successfully, and outputs it if it is executed successfully, and performs SQL repair if it fails; the electronic device calls the SQL executor to execute the repaired target SQL statement and determines whether it is executed successfully; it outputs it if it is executed successfully, and discards it if it fails.

[0121] In order to lower the technical threshold for data access, improve user experience, enhance interactive friendliness, and have significant advantages, so as to enable the comprehensive application of big models in SQL field scenarios, based on the above embodiments, in the embodiment of the present application, the task of scheduling the big model to execute the target intent includes:

[0122] Scheduling the large model so that the large model determines target parameters used when performing the task of the target intent;

[0123] The large model is controlled to process the target text according to the target parameters.

[0124] In an embodiment of the present application, when the electronic device trains the big model, it trains the big model separately for each intent, that is, when the big model executes the task corresponding to each intent, it will operate using the parameters corresponding to the intent.

[0125] Based on this, in an embodiment of the present application, when an electronic device schedules a large model to process a target text, the electronic device can schedule the large model so that the large model determines the target parameters used when performing a task with target intent; and controls the large model to process the target text according to the target parameters.

[0126] In addition, in an embodiment of the present application, the target text can also be the text of a multi-task hybrid application, that is, the target text corresponds to multiple intents. In hybrid applications, the embodiment of the present application first decides which set of data processing prompts and parameters to use based on the multiple intents corresponding to the target text. Different intents feed different data processing prompt texts and different parameters. In addition, in this section, each task will use a hybrid retrieval strategy to provide examples for the large model to improve the understanding generation capabilities of the large model.

[0127] That is, in the embodiment of the present application, if the target text is a text with multiple intents, the electronic device determines the relationship between each intent, determines the intent to be finally realized, takes the data processing prompt text corresponding to the intent as input, and adds examples and database table structure information, task constraints, business constraints, etc. corresponding to other intents to the data processing prompt text. The electronic device calls the large model to execute the task corresponding to each intent according to the parameters corresponding to each intent.

[0128] Figure 8 A schematic diagram of the data processing process of the multi-intention text provided in the embodiment of the present application, such as the Figure 8As shown, the process includes: the electronic device performs text classification on the user question (target question) to determine that the user question has multiple intentions; the electronic device determines the data processing prompt text corresponding to the intention to be finally achieved, and constructs a similar example; the electronic device inputs the data processing prompt text, similar examples and target text (question) into the large model.

[0129] In a possible implementation, if the target text of the multi-intent is "What is the sales volume of Qingdao Marketing Center", the input of the large model is:

[0130]

[0131]

[0132] Table 4

[0133] Table 4 is a multi-intention large model input provided in an embodiment of the present application. As shown in Table 4, the input includes task instructions, database table structure information, task constraints + business constraints, examples, and target text in natural language format.

[0134] In an embodiment of the present application, the electronic device can retrieve historical texts similar to the target text from the historical case library through a pre-configured hybrid retrieval model, and input the historical texts and corresponding historical SQL statements as examples into the large model. The hybrid retrieval model can be obtained by combining BM25, Embedding, and Rerank.

[0135] Specifically, BM25 is called to filter the first candidate historical text with similarity to the target text from the historical case library through keywords; Embedding is called to filter the second candidate historical text with similarity to the target text from the historical case library through vectors; Rerank is then called to rearrange the first candidate historical text and the second candidate historical text, and the candidate historical text with the greatest similarity obtained by rearrangement and the corresponding historical SQL statement are used as examples.

[0136] Fig. 9 A schematic diagram of the hybrid search process provided in the embodiment of the present application, such as the Fig. 9 As shown, the process includes performing keyword search and vector search according to user questions to obtain search results; rearranging the search results and outputting examples.

[0137] Based on the above embodiments, Fig.10 A schematic diagram of a data processing device based on a large model provided in an embodiment of the present application, the device comprising:

[0138] The intention recognition module 1001 is used to receive a target text input by a user and determine a target intention of the target text; wherein the target intention is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display;

[0139] Processing module 1002 is used to obtain the pre-configured data processing prompt text corresponding to the target intent, and input the prompt text and the target text into the big model; wherein the big model is obtained by mixed training based on sample data of multiple tasks; schedule the big model to execute the task of the target intent, and obtain the processing result output by the big model.

[0140] In a possible implementation, the processing module 1002 is also used to obtain, for each intent, a pre-configured data processing prompt text corresponding to the intent, wherein the data processing prompt text includes task instructions, task constraints and business constraints corresponding to the target intent; input the data processing prompt text, pre-configured sample data and corresponding sample processing results into the big model; and fine-tune the big model according to the lora fine-tuning algorithm to determine the parameters used by the big model when executing the task corresponding to the intent.

[0141] In a possible implementation, if the intent is Text2SQL or SQL2Text, the data processing prompt text also includes database table structure information.

[0142] In a possible implementation, the processing module 1002 is further used to determine a target feature vector corresponding to the target text according to a preset embedding representation algorithm; obtain each first feature vector corresponding to the saved summary description of each database table, and determine a first preset number of target database tables that are most similar to the target text according to each first feature vector and the target feature vector; obtain each second feature vector corresponding to each field description of the saved target database table, and determine a second preset number of target fields that are most similar to the target text according to the second feature vector and the target feature vector; and determine the table names of the first preset number of target database tables and the field information of the second preset number of target fields as the database table structure information.

[0143] In a possible implementation, the processing module 1002 is further used to obtain the target SQL statement output by the big model if the target intent is Text2SQL; execute the target SQL statement; if the execution fails, input the pre-configured SQL repair prompt text and the target SQL statement into the big model so that the big model repairs the target SQL statement; and obtain the repaired target SQL statement output by the big model.

[0144] In a possible implementation, the processing module 1002 is further configured to execute the repaired target SQL statement; if the execution fails, output response information indicating the processing failure.

[0145] In a possible implementation, the processing module 1002 is specifically used to schedule the large model so that the large model determines the target parameters used when executing the task of the target intention; and controls the large model to process the target text according to the target parameters.

[0146] Based on the above embodiments, the present application also provides an electronic device, Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, it includes: a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through the communication bus 1104;

[0147] The memory 1103 stores a computer program. When the program is executed by the processor 1101, the processor 1101 executes the steps of the large model-based data processing method provided in the above-mentioned embodiments.

[0148] Since the principle of solving the problem by the above electronic device is similar to the data processing method based on the large model, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0149] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0150] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0151] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based data processing method provided in the above embodiments.

[0152] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data processing method based on a large model, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data processing method based on a large model, characterized in that: The method comprises: Receive a target text input by a user, and determine a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display; Obtaining a pre-configured data processing prompt text corresponding to the target intent, and inputting the prompt text and the target text into a large model; wherein the large model is obtained by mixed training based on sample data of multiple tasks; The large model is scheduled to execute the task of the target intention, and the processing result output by the large model is obtained.

2. The method according to claim 1, characterized in that The training process of the large model includes: For each intent, obtain the pre-configured data processing prompt text corresponding to the intent, wherein the data processing prompt text includes the task instructions, task constraints and business constraints corresponding to the target intent; input the data processing prompt text, pre-configured sample data and corresponding sample processing results into the big model; fine-tune the big model according to the lora fine-tuning algorithm to determine the parameters used by the big model when executing the task corresponding to the intent.

3. The method according to claim 2, characterized in that If the intent is Text2SQL or SQL2Text, the data processing prompt text also includes database table structure information.

4. The method according to claim 3, characterized in that The process of determining the database table structure information includes: Determine the target feature vector corresponding to the target text according to a preset embedding representation algorithm; Obtaining each first feature vector corresponding to the saved summary description of each database table, and determining a first preset number of target database tables that are most similar to the target text according to each first feature vector and the target feature vector; Obtaining each second feature vector corresponding to each field description of the target database table that is saved, and determining a second preset number of target fields that are most similar to the target text according to the second feature vector and the target feature vector; The table names of the first preset number of target database tables and the field information of the second preset number of target fields are determined as the database table structure information.

5. The method according to claim 1, characterized in that If the target intent is Text2SQL, the processing result of obtaining the large model output includes: Obtain the target SQL statement output by the large model; The method further comprises: Execute the target SQL statement; If the execution fails, the pre-configured SQL repair prompt text and the target SQL statement are input into the large model so that the large model repairs the target SQL statement; Obtain the repaired target SQL statement output by the large model.

6. The method according to claim 5, characterized in that The method further comprises: Execute the repaired target SQL statement; If the execution fails, the response information of processing failure will be output.

7. The method according to claim 1, characterized in that The task of scheduling the large model to execute the target intent includes: Scheduling the large model so that the large model determines target parameters used when performing the task of the target intent; The large model is controlled to process the target text according to the target parameters.

8. A data processing device based on a large model, characterized in that: The device comprises: An intention recognition module, used to receive a target text input by a user and determine a target intention of the target text; wherein the target intention is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display; A processing module is used to obtain a pre-configured data processing prompt text corresponding to the target intent, and input the prompt text and the target text into a large model; wherein the large model is obtained by mixed training based on sample data of multiple tasks; the large model is scheduled to execute the task of the target intent, and obtain the processing result output by the large model.

9. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the large model-based data processing method as claimed in any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the large model-based data processing method as described in any one of claims 1 to 7.

Images