Data processing method and device based on large model, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device based on a large model, equipment and a medium. In the embodiment of the invention, the electronic equipment performs multi-task mixed training on a large amount of sample data and the corresponding natural language description to obtain the large model, so that the large model is trained to complete multiple types of tasks. In the reasoning stage, the electronic equipment can perform text classification on the received target text so as to determine the to-be-executed target task type, and meanwhile, based on the dynamic example recall strategy, the cue word of the target task type is determined, and the cue word and the target text are input into the large model, so that the understanding generation capability of the large model is improved, and the user experience is improved. According to the method, the technical threshold of data access is greatly reduced, the user experience is improved, the interaction friendliness is enhanced, and the like, so that a large model is comprehensively applied to scenes in the field of SQL (Structured Query Language).

Description

[0001] This application claims priority to the Chinese Patent Application No. 202411834749.6, filed on December 12, 2024, entitled “A data processing method, device and equipment based on a large model and a medium”, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, in particular to a data processing method, device and equipment based on a large model and a medium. BACKGROUND

[0003] With the development of artificial intelligence technology, the use scenarios of large models are increasingly widespread. However, in the current use scenarios, only the large model is applied for Text2SQL query. For example, after the large model receives the question text “query the GDP of Qingdao city”, the large model generates SQL, and the electronic device gives the user a reply answer according to the SQL. This results in that the large model only plays a partial role and does not fully play other capabilities of the large model in SQL application, i.e., the application efficiency of the large model is low. SUMMARY

[0004] The present application provides a data processing method, device and equipment based on a large model and a medium, to solve the problem that the large model only plays a partial role and does not fully play other capabilities of the large model in SQL application, i.e., the application efficiency of the large model is low.

[0005] In a first aspect, the embodiments of the present application provide a data processing method based on a large model, the method comprising:

[0006] receiving a target text input by a user and determining a target intention of the target text; wherein the target intention is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display;

[0007] obtaining a data processing prompt text corresponding to the target intention pre-configured, inputting the prompt text and the target text into a large model; wherein the large model is obtained by mixed training of sample data based on multiple tasks;

[0008] scheduling the large model to execute a task of the target intention, and obtaining a processing result output by the large model.

[0009] In a second aspect, the embodiments of the present application further provide a data processing device based on a large model, the device comprising:

[0010] An intent recognition module is configured to receive target text input by a user and determine a target intent of the target text, wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display.

[0011] A processing module is configured to obtain a data processing prompt text corresponding to the target intent pre-configured, input the prompt text and the target text into a large model, wherein the large model is obtained by mixed training of sample data based on multiple tasks, schedule the large model to perform a task of the target intent, and obtain a processing result output by the large model.

[0012] In a third aspect, an electronic device is provided, and the electronic device includes at least a processor and a memory. The processor is configured to implement the steps of the data processing method based on the large model according to any of the above aspects when executing a computer program stored in the memory.

[0013] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to implement the steps of the data processing method based on the large model according to any of the above aspects when executed by a processor.

[0014] In the embodiments of the present application, the electronic device obtains the large model by performing multi-task mixed training on a large amount of sample data and corresponding natural language descriptions, and realizes training of the large model to complete multiple types of tasks. In the inference stage, the electronic device can perform text classification on the received target text to determine the target task type to be executed, and determine the prompt word of the target task type based on a dynamic example recall strategy, input the prompt word and the target text into the large model, and improve the understanding and generation ability of the large model. The present application greatly reduces the technical threshold of data access, improves user experience, enhances interaction friendliness, and has significant advantages in the above aspects, so that the large model is comprehensively applied in the SQL field. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A data processing process based on a large model is provided for the embodiments of the present application.

[0017] Figure 2 A function diagram of an electronic device is provided for the embodiments of the present application.

[0018] Figure 3 An intended fine-tuning process diagram provided for an embodiment of the present application;

[0019] Figure 4 A plurality of intended fine-tuning process diagrams provided for an embodiment of the present application;

[0020] Figure 5 A large model fine-tuning diagram provided for an embodiment of the present application;

[0021] Figure 6 A data processing flowchart provided for an embodiment of the present application;

[0022] Figure 7 A Text2SQL intended data processing flowchart provided for an embodiment of the present application;

[0023] Figure 8 A data processing process diagram of multi-intended text provided for an embodiment of the present application;

[0024] Figure 9 A hybrid retrieval flowchart provided for an embodiment of the present application;

[0025] Figure 10 A data processing device structure diagram based on a large model provided for an embodiment of the present application;

[0026] Figure 11 An electronic device structure diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0028] In order to reduce the technical threshold of data access, improve user experience, enhance interaction friendliness, etc., and have significant advantages in the comprehensive application of large models in SQL field scenarios, the present application provides a data processing method, device, equipment and medium based on a large model.

[0029] In the embodiment of the present application, the electronic device receives a target text input by a user, determines a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display; a data processing prompt text corresponding to the target intent is obtained, and the prompt text and the target text are input into a large model; wherein the large model is obtained based on mixed training of sample data of multiple tasks; the large model is scheduled to perform the task of the target intent, and a processing result output by the large model is obtained.

[0030] Figure 1 A data processing process based on a large model is provided in the embodiment of the present application, and the process includes:

[0031] S101: receiving a target text input by a user, and determining a target intent of the target text; wherein the target intent is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display.

[0032] The data processing method based on a large model provided in the embodiment of the present application is applied to an electronic device, which can be a PC or a server.

[0033] In the embodiment of the present application, the electronic device performs mixed training on the large model based on sample data of multiple tasks, so that the large model can complete multiple tasks, greatly reduces the technical threshold of data access, and has significant advantages in improving user experience and enhancing interactive friendliness, so that the large model is comprehensively applied in the SQL field.

[0034] Therefore, the user can input the target text to be processed to the electronic device in the form of input text, and the electronic device dynamically constructs a prompt text and schedules the large model according to the task to be performed by the large language model for the target text after receiving the target text.

[0035] Specifically, the electronic device performs intent recognition on the target text to determine the target intent corresponding to the target text, wherein one intent corresponds to one task. The electronic device can use multiple methods to perform intent recognition on the target text, including but not limited to: inputting the target text into a trained intent recognition model, scheduling the intent recognition model to perform intent recognition on the target text, and determining the target intent of the target text; or performing keyword detection on the target text, determining the intent corresponding to the target keyword as the target intent of the target text based on the detected target keyword and the preconfigured correspondence between the keyword and the intent; or inputting the target text into a large language model, scheduling the large language model to determine and output the target intent of the target text. The intent recognition model can also be a classification model, etc., which is not limited here.

[0036] In the embodiments of the present application, the target intent of the target text determined by the electronic device is one of library table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display.

[0037] Among them, the library table management includes but is not limited to database management and table management. The database management includes but is not limited to creating a database table, deleting a database table, and modifying the name of a data table, etc. For example, the target text input by the user is "create ai_test database", and the electronic device determines that the intent of the target text is library table management. The table management includes but is not limited to creating a table field, modifying a table field, and deleting a table field, etc. Among them, the table field is created by previously manually assigning a type, increasing a field attribute introduction, and designing a table field name to be automatically created. For example, the target text input by the user is "create a store sales data table, the fields include year, month, day, complete date, store name, first-level area, second-level area, first-level channel, store daily sales, store daily fixed cost, daily initial inventory, and daily final inventory, and generate appropriate indexes for the table fields", and the electronic device determines that the target intent of the target text is library table management.

[0038] Text2SQL is to generate an SQL statement according to the target text input by the user. For example, the target text is "how many people come into the store every day?", and the electronic device determines that the target intent is Text2SQL.

[0039] In SQL2Text, the target text input by the user is an SQL statement, and the electronic device needs to convert the SQL statement into a natural language text. It can also convert the SQL. For example, the target text is "SELECT * FROM user WHERE age>30", and the electronic device determines that the target intent is SQL2Text.

[0040] In a possible implementation, if the target text is "SELECT * FROM user WHERE age>30", the processing result determined by the electronic device includes but is not limited to statement interpretation and statement optimization.

[0041] Among them, the statement interpretation can be:

[0042] The meaning of this SQL statement is to extract all user records with an age greater than 30 from the users table.

[0043] This SQL statement is a typical query statement used to retrieve data from a database with specific conditions. Specifically, it selects all columns (indicated by the asterisk *) from the users table, but only includes records where the age is greater than 30. This query can be used for various purposes, such as if a user wants to view detailed information of all adult users or perform some age-based data analysis.

[0044] The components of this statement explain its functionality:

[0045] SELECT * indicates that all columns of data are selected.

[0046] FROM users specifies which table to extract data from, in this case, the users table

[0047] WHERE age > 30 is a conditional clause that limits the query results to only include records where the age is greater than 30.

[0048] The statement can be optimized as: SELECT id, name, email, age FROM user WHERE age > 30.

[0049] Data generation can be used to generate training data for large models or some simulated data for model training. This includes but is not limited to generating data and modifying existing database table structures, such as adding columns, etc. For example, if the target text is "Help me generate 15 pieces of near-real data information, the first-level region includes Southwest, Northwest, etc., among which the first-level region Southwest includes the second-level region cities of Degree, Guiyang, Chongqing, and the first-level region Northwest includes the second-level region cities of Lanzhou, Xi'an, etc.", the electronic device determines that the target intent is data generation; if the target text is "Add a column of store visitor number, column name in English, and randomly fill in the existing data", the electronic device determines that the target intent is data generation.

[0050] Table field optimization is to convert non-standard table fields into the best standard syntax of the target database system by deeply understanding the field names and field values of the source database system using a large model. For example, if the target text is "Help me optimize the ai_test table field", the electronic device determines that the target intent is table field optimization.

[0051] In the embodiments of the present application, the content that can be optimized by table field optimization includes but is not limited to: pinyin to professional English, professional English according to annotations, incorrect words to correct words, and type optimization, etc., all of which are converted into words that are easier for the large model to understand.

[0052] The chart display is used to support the chart display of numerical data in the database, and can support column chart, line chart, pie chart, rose chart, bar chart and the like. The data distribution in the database can be displayed at a glance. For example, if the target text is "horizontal coordinate is region, vertical coordinate is 100 million yuan, and product sales are displayed by column chart", the electronic device determines that the target intent is chart display; if the target text is "displayed by bar chart", the electronic device determines that the target intent is chart display.

[0053] In S102, a pre-configured data processing prompt text corresponding to the target intent is obtained, and the prompt text and the target text are input into a large model; the large model is obtained based on mixed training of sample data of multiple tasks; the large model is scheduled to perform a task of the target intent, and a processing result output by the large model is obtained.

[0054] In the embodiment of the present application, when training the large model, different data processing prompt texts are used for different intents. The large model can be a large language model such as GPT-3, Wenyan-yixin, etc., or a multi-modal large model such as GPT-4, etc., which is not limited here.

[0055] Based on this, in the embodiment of the present application, after the electronic device determines the target intent of the target text, the electronic device further obtains a pre-configured data processing prompt text corresponding to the target intent, inputs the prompt text and the target text into the large model, so that the large model processes the target text according to the data processing prompt text. The large model is obtained based on mixed training of sample data of multiple tasks.

[0056] In the embodiment of the present application, the electronic device schedules the large model to perform a task corresponding to the target intent through the data processing prompt text, so that the large model determines and outputs a processing result for the target text.

[0057] Each intent corresponds to a data prompt text, which includes but is not limited to task instructions, task constraints and business constraints. The task instructions are used to prompt the large model to perform which task corresponding to the intent, such as the task instruction corresponding to the SQL2Text intent, which is "you are now a database expert, and the task now is to generate corresponding explanation and optimized SQL according to the user input SQL". The task constraints and business constraints are used to constrain the processing process of the large model, such as the task constraints and business constraints corresponding to the SQL2Text intent, which are "## always remember that the output result must be a SQL statement; ## try not to use * when querying all, and need to use the field name in the database; ## the SQL must give the corresponding explanation, and the SQL cannot be optimized, then output as it is".

[0058] Figure 2 The function schematic diagram of the electronic device provided in the embodiment of the present application is shown in the figure. Figure 2 As shown in the figure, the electronic device performs text fusion and text classification on the user question, and can implement the tasks of library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display.

[0059] In the embodiment of the present application, the electronic device obtains a large model by performing multi-task mixed training on a large amount of sample data and corresponding natural language descriptions, and realizes that the large model can complete multiple types of tasks. In the inference stage, the electronic device can perform text classification on the received target text to determine the target task type to be executed, and based on a dynamic example recall strategy, determine the prompt word of the target task type, input the prompt word and the target text into the large model, and improve the understanding and generation capabilities of the large model. The present application greatly reduces the technical threshold of data access, improves user experience, enhances interaction friendliness, and has significant advantages in other aspects, so that the large model can be comprehensively applied in the SQL field.

[0060] In order to reduce the technical threshold of data access, improve user experience, enhance interaction friendliness, and have significant advantages in other aspects, so that the large model can be comprehensively applied in the SQL field, on the basis of the above embodiment, in the embodiment of the present application, the training process of the large model includes:

[0061] For each intent, a pre-configured data processing prompt text corresponding to the intent is obtained, wherein the data processing prompt text includes task instructions, task constraints and business constraints corresponding to the target intent; the data processing prompt text, pre-configured sample data and corresponding sample processing results are input into the large model; according to the lora fine-tuning algorithm, the large model is fine-tuned to determine the parameters used by the large model when executing the task corresponding to the intent.

[0062] In order to make the large model more suitable for implementing multiple tasks, in the embodiment of the present application, the electronic device will use sample data of multiple tasks to train the large model. Specifically, the present application uses the lora method for training, and there is no innovation in the lora method itself, but in order to improve the performance of the large model, the present application introduces a special LoRA module for each task to improve the performance of the large model in the multi-task prediction task.

[0063] Specifically, for each intent, the electronic device obtains a pre-configured data processing prompt text corresponding to the intent, wherein the data processing prompt text includes a task instruction, a task constraint and a business constraint corresponding to the target intent. The electronic device inputs the data processing prompt text, the pre-configured sample data and the corresponding sample processing result into the large model, and fine-tunes the large model according to the lora fine-tuning algorithm to determine the parameters used by the large model when executing the task corresponding to the intent.

[0064] Figure 3 A fine-tuning process diagram of one intent provided by an embodiment of the present application is shown in the figure. Figure 3 As shown in the figure, when training the large model, the parameters of the large model need to be updated, which can be represented as W0+ΔW, wherein W0 is the initialized parameter and ΔW is the parameter to be updated. For the lora fine-tuning algorithm, only ΔW needs to be fine-tuned.

[0065] Specifically, assume that the parameters of the large model W0∈R d×k His update can be represented as:

[0066] W0+ΔW=W0+BA,B∈R d×r ,A∈R r×k

[0067] Wherein W0 is fixed and only A and B are training parameters, and the rank r << min(d, k).

[0068] Figure 4 A fine-tuning process diagram of multiple intents provided by an embodiment of the present application is shown in the figure. Figure 4 As shown in the figure, a low-rank matrix is used as the parameter of the large model to learn the individualized information of each intent. It can be expressed as follows, t represents the tth task:

[0069] y=W+ΔW t =W+BA

[0070] W represents the pre-training parameter, A and B are training parameters, t represents the task corresponding to the tth intent, and y represents all parameters of the large model.

[0071] Figure 5 A large model fine-tuning diagram provided by an embodiment of the present application is shown in the figure. Figure 5 As shown in the figure, the existing base large model has the problem of low accuracy in the face of current Text2SQL and SQL2Text tasks. To solve this problem, the present application carries out large model fine-tuning and prompt word engineering, including calling the base model for single task training, and through dynamic example recall, so that the large model can be used for library table management, Text2SQL, SQL2Text, data generation, table field optimization and chart display.

[0072] In order to reduce the technical threshold of data access, improve user experience, enhance interaction friendliness, etc., it has significant advantages, so that the large model can be applied in the SQL field scene, and on the basis of the above embodiments, in the embodiment of the present application, if the intention is Text2SQL or SQL2Text, the data processing prompt text further includes database table structure information.

[0073] In the embodiment of the present application, for Text2SQL or SQL2Text, in order to make the large language model better process the target text, the database table structure information is further included in the data processing prompt text corresponding to Text2SQL or SQL2Text.

[0074] Among them, the database table structure information includes but is not limited to the name of the database table, the name of the field, and the value corresponding to the field.

[0075] In order to reduce the technical threshold of data access, improve user experience, enhance interaction friendliness, etc., it has significant advantages, so that the large model can be applied in the SQL field scene, and on the basis of the above embodiments, in the embodiment of the present application, the determination process of the database table structure information includes:

[0076] According to a preset embedding representation algorithm, a target feature vector corresponding to the target text is determined;

[0077] Obtain each first feature vector corresponding to the saved summary description of each database table, and determine a first preset number of target database tables most similar to the target text according to the first feature vector and the target feature vector.

[0078] Obtain each second feature vector corresponding to the saved field description of the target database table, and determine a second preset number of target fields most similar to the target text according to the second feature vector and the target feature vector.

[0079] The table name of the first preset number of target database tables and the field information of the second preset number of target fields are determined as the database table structure information.

[0080] In the embodiment of the present application, when determining the database table structure information, the electronic device can determine based on table recall, field recall, and enumeration value recall.

[0081] Specifically, the electronic device pre-describes each database table in the database, describes the purpose of each database table, and explains which fields the user can inquire about for each database table. At the same time, the electronic device describes the fields of each database table. The electronic device uses vector models such as BGE and BCE to perform Embedding coding on the generated summary description, field description, and enumeration value, obtains and saves a first feature vector corresponding to each summary description, a second feature vector corresponding to each field description, and a third feature vector corresponding to each enumeration value.

[0082] Based on this, when determining the data table structure information corresponding to the target text, the electronic device determines a target feature vector corresponding to the target text according to a preset embedding representation algorithm; obtains each first feature vector corresponding to the summary description of each database table saved, and determines a first preset number of target database tables most similar to the target text according to each first feature vector and the target feature vector. The electronic device obtains each second feature vector corresponding to the field description of each target database table saved, and determines a second preset number of target fields most similar to the target text according to the second feature vector and the target feature vector. The electronic device determines the table name of the first preset number of target database tables and the field information of the second preset number of target fields as the data table structure information.

[0083] Among them, the field information includes but is not limited to the field name, the field description, and the value corresponding to the field.

[0084] In addition, in the embodiment of the present application, if the target field is an enumeration type field, the electronic device can also obtain each enumeration value corresponding to the target field and the corresponding third feature vector saved, and determine a second preset number of target enumeration values most similar to the target text according to the third feature vector and the target feature vector. The electronic device determines the table name of the first preset number of target database tables, the field name of the second preset number of target fields, and the third preset number of target enumeration values as the data table structure information.

[0085] Among them, in the embodiment of the present application, the first preset number, the second preset number and the third preset number can be configured according to user demand, but the value cannot exceed the maximum value that can be selected. That is, the first preset number cannot exceed the total number of database tables contained in the database, the second preset number cannot exceed the total number of fields contained in the target database table, and the third preset number cannot exceed the total number of enumeration values contained in the target field.

[0086] In a possible manner, the electronic device can use a cosine similarity algorithm to perform similarity calculation of vectors, such as Res = cosine (Embedding (target question), Embedding (abstract description / field description / enumeration value)).

[0087] It should be noted that in the embodiments of the present application, when the target fields are determined, the second preset number of target fields can be selected from each target database table respectively, or the second preset number of target fields can be selected. Generally, the second preset number of target fields is selected from each target database table respectively.

[0088] Specifically, after the electronic device completes the similarity calculation, the electronic device obtains Top-K target database tables higher than the threshold value, and obtains Top-N target fields of each target database table to constitute the structure information of the database. The purpose is to enhance the generalization of the large model after training, improve the discrimination ability of the large model for similar fields, save the length of the Token input of the large model, and reduce the influence of redundant data on the input of the large model.

[0089] On the basis of the above-mentioned embodiments, in the embodiments of the present application, the electronic device can input the data processing prompt text and the target question into the large model for processing.

[0090] Figure 6 The data processing flowchart provided by the embodiments of the present application is as shown in the table 1. Figure 6 The process includes: the electronic device performs text classification on the user question (target question) to determine that the intention of the user question is Text2SQL; the electronic device performs field recall, table recall and enumeration value recall to determine the database table structure information; and the electronic device inputs the database table structure information, business constraints and instructions into the large model.

[0091] In a possible implementation, the input of the large model can be:

[0092]

[0093] Table 1

[0094] Table 1 is the input of the Text2SQL intention of the large model provided by the embodiments of the present application, as shown in the table 1, the input includes task instructions, database table structure information, task constraints + business constraints and natural language format target text.

[0095] In another possible implementation, the input of the large model can be:

[0096]

[0097] Table 2

[0098] Table 2 shows the large model input of the SQL2Text intent provided in the embodiments of this application. As shown in Table 2, the input includes task instructions, database table structure information, task constraints + business constraints, and target text in SQL format.

[0099] It is worth noting that, apart from the Text2SQL and SQL2Text intents, when processing the target text of other intents, it is not necessary to recall the database table structure information. Only the data processing prompt text containing the task instructions, task constraints, and business constraints, as well as the target text, need to be input into the large language model.

[0100] Furthermore, in other intents, different data processing prompt texts are constructed according to different intents for large model training, and a set of data processing prompt texts is designed for each intent for training.

[0101] To lower the technical barrier to data access, improve user experience, and enhance interactivity, thereby enabling the comprehensive application of large models in SQL scenarios, based on the above embodiments, in this embodiment, if the target intent is Text2SQL, the processing result of obtaining the output of the large model includes:

[0102] Obtain the target SQL statement output by the large model;

[0103] The method further includes:

[0104] Execute the target SQL statement;

[0105] If the execution fails, the pre-configured SQL repair prompt text and the target SQL statement will be input into the large model, so that the large model can repair the target SQL statement.

[0106] Obtain the repaired target SQL statement output by the large model.

[0107] In this embodiment, after the large language model outputs the target SQL statement, the electronic device acquires the target SQL statement and sends it to the data source for execution. If the execution is successful, there are no syntax errors, and the statement is output. If the execution identifies a syntax error, the electronic device constructs a new prompt for the large model to perform SQL error correction.

[0108] Specifically, the electronic device obtains the target SQL statement output by the large model and executes it. If the execution fails, the electronic device inputs the pre-configured SQL repair prompt text and the target SQL statement into the large model, causing the large model to repair the target SQL statement and obtain the repaired target SQL statement output by the large model.

[0109] The SQL repair prompt text includes, but is not limited to, task instructions, database table structure information, task constraints + business constraints, and examples.

[0110] In one possible implementation, when the electronic device invokes a large language model for SQL repair, the input to the large model can be:

[0111]

[0112]

[0113] Table 3

[0114] Table 3 shows the large model input for SQL repair provided in the embodiments of this application. As shown in Table 3, the input includes task instructions, database table structure information, task constraints + business constraints, examples, and target SQL statements.

[0115] To lower the technical barriers to data access, improve user experience, and enhance interactivity, thereby enabling the comprehensive application of large models in SQL scenarios, the method in this embodiment further includes, based on the above embodiments:

[0116] Execute the repaired target SQL statement;

[0117] If the execution fails, output a response message indicating that the process failed.

[0118] In this embodiment, after the electronic device calls the large model to repair the target SQL statement, the electronic device will execute the repaired target SQL statement again. If the execution of the repaired target SQL statement still fails, the electronic device will output a processing failure response message.

[0119] Specifically, after correcting the syntax of the target SQL statement, the electronic device schedules the SQL executor to run the code of the repaired target SQL statement. If the execution is successful, the repaired target SQL statement is output as a reference. If the SQL execution continues to encounter errors, it is discarded and no longer used, and a response message indicating that the SQL could not be generated is sent back to the user.

[0120] Figure 7 This is a data processing flowchart of the Text2SQL intent provided in the embodiments of this application, as shown below. Figure 7As shown, the electronic device calls the Large Language Model (LLM) to perform Text2SQL on the target text to obtain the target SQL statement; the electronic device calls the SQL executor to execute the target SQL statement; the electronic device determines whether the target SQL statement is executed successfully, and outputs it if it is executed successfully, and performs SQL repair if it fails; the electronic device calls the SQL executor to execute the repaired target SQL statement and determines whether it is executed successfully; if it is executed successfully, it is output, and if it fails, it is discarded.

[0121] To lower the technical barriers to data access, improve user experience, and enhance interactivity, thereby enabling the comprehensive application of large models in SQL scenarios, based on the above embodiments, in this embodiment, the task of scheduling the large model to execute the target intent includes:

[0122] The large model is scheduled to determine the target parameters used when performing the task with the target intent;

[0123] The large model is controlled to process the target text according to the target parameters.

[0124] In the embodiments of this application, when the electronic device trains the large model, it trains the large model separately for each intent. That is, when the large model executes the task corresponding to each intent, it will run using the parameters corresponding to that intent.

[0125] Based on this, in the embodiments of this application, when the electronic device schedules the large model to process the target text, the electronic device can schedule the large model to determine the target parameters used when performing the task of the target intent; and control the large model to process the target text according to the target parameters.

[0126] Furthermore, in this embodiment, the target text can also be text used in multi-task hybrid applications, meaning the target text corresponds to multiple intents. In hybrid applications, this embodiment first determines which set of data processing prompts and parameters to use based on the multiple intents corresponding to the target text. Different intents are fed different data processing prompt texts and different parameters. Additionally, in this section, each task will use a hybrid retrieval strategy to provide examples for the large model, thereby improving the large model's understanding and generation capabilities.

[0127] That is, in this embodiment, if the target text is text with multiple intents, the electronic device determines the relationship between each intent, determines the final intent to be implemented, takes the data processing prompt text corresponding to that intent as input, and adds examples and other database table structure information, task constraints, business constraints, etc., corresponding to other intents to the data processing prompt text. The electronic device calls the large model to execute the task corresponding to each intent according to the parameters corresponding to each intent.

[0128] Figure 8 This is a schematic diagram of the data processing process for multi-intent text provided in the embodiments of this application, as shown below. Figure 8 As shown, the process includes: the electronic device classifies the user's question (target question) into text and determines that the user's question has multiple intents; the electronic device determines the data processing prompt text corresponding to the final intent to be achieved and constructs similar examples; the electronic device inputs the data processing prompt text, similar examples, and target text (question) into a large model.

[0129] In one possible implementation, if the target text of the multi-intent is "What is the sales volume of the Qingdao Marketing Center?", then the input to the large model is:

[0130]

[0131]

[0132] Table 4

[0133] Table 4 shows the input of the large model with multiple intents provided in the embodiments of this application. As shown in Table 4, the input includes task instructions, database table structure information, task constraints + business constraints, examples, and target text in natural language format.

[0134] In this embodiment, the electronic device can retrieve historical text similar to the target text from a historical case library using a pre-configured hybrid retrieval model, and input the historical text and corresponding historical SQL statements as examples into a large model. This hybrid retrieval model can be obtained by combining BM25, Embedding, and Rerank.

[0135] Specifically, BM25 is called to filter the first candidate historical text with the same similarity to the target text from the historical case library using keywords; Embedding is called to filter the second candidate historical text with the same similarity to the target text from the historical case library using vectors; Rerank is then called to rearrange the first and second candidate historical texts, and the candidate historical text with the highest similarity obtained from the rearrangement and the corresponding historical SQL statement are used as examples.

[0136] Figure 9This is a schematic diagram of the hybrid retrieval process provided in the embodiments of this application, as shown below. Figure 9 As shown, the process includes keyword retrieval and vector retrieval based on the user's question to obtain retrieval results; rearranging the retrieval results and outputting examples.

[0137] Based on the above embodiments, Figure 10 A schematic diagram of a data processing device based on a large model is provided in this application embodiment. The device includes:

[0138] The intent recognition module 1001 is used to receive target text input by the user and determine the target intent of the target text; wherein, the target intent is one of the following: database table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display;

[0139] The processing module 1002 is used to obtain the pre-configured data processing prompt text corresponding to the target intent, input the prompt text and the target text into a large model; wherein, the large model is obtained by training based on sample data of multiple tasks; schedule the large model to execute the task of the target intent, and obtain the processing result output by the large model.

[0140] In one possible implementation, the processing module 1002 is further configured to, for each intent, obtain a pre-configured data processing prompt text corresponding to that intent, wherein the data processing prompt text includes the task instruction, task constraints, and business constraints corresponding to the target intent; input the data processing prompt text, pre-configured sample data, and corresponding sample processing results into the large model; and fine-tune the large model according to the LoRa fine-tuning algorithm to determine the parameters used by the large model when executing the task corresponding to the intent.

[0141] In one possible implementation, if the intent is Text2SQL or SQL2Text, the data processing prompt text also includes database table structure information.

[0142] In one possible implementation, the processing module 1002 is further configured to: determine the target feature vector corresponding to the target text according to a preset embedding representation algorithm; obtain each first feature vector corresponding to the summary description of each saved database table, and determine a first preset number of target database tables most similar to the target text according to each first feature vector and the target feature vector; obtain each second feature vector corresponding to each field description of the saved target database table, and determine a second preset number of target fields most similar to the target text according to the second feature vector and the target feature vector; and determine the table names of the first preset number of target database tables and the field information of the second preset number of target fields as the database table structure information.

[0143] In one possible implementation, the processing module 1002 is further configured to: if the target intent is Text2SQL, obtain the target SQL statement output by the large model; execute the target SQL statement; if the execution fails, input the pre-configured SQL repair prompt text and the target SQL statement into the large model, so that the large model repairs the target SQL statement; and obtain the repaired target SQL statement output by the large model.

[0144] In one possible implementation, the processing module 1002 is further configured to execute the repaired target SQL statement; if the execution fails, a processing failure response message is output.

[0145] In one possible implementation, the processing module 1002 is specifically configured to schedule the large model, enabling the large model to determine the target parameters used when performing the task of the target intent; and to control the large model to process the target text according to the target parameters.

[0146] Based on the above embodiments, this application also provides an electronic device. Figure 11 This application provides a schematic diagram of an electronic device structure, such as... Figure 11 As shown, it includes: processor 1101, communication interface 1102, memory 1103 and communication bus 1104, wherein processor 1101, communication interface 1102 and memory 1103 communicate with each other through communication bus 1104.

[0147] The memory 1103 stores a computer program, which, when executed by the processor 1101, causes the processor 1101 to perform the steps of the data processing method based on a large model as provided in the above embodiments.

[0148] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the data processing method based on large models, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated details will not be repeated.

[0149] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. Communication interface 1102 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0150] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital instruction processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0151] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to implement the data processing method steps based on a large model as provided in the above embodiments.

[0152] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the data processing method based on large models, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be described again.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method based on a large model, characterized in that, The method includes: Receive target text input by the user and determine the target intent of the target text; wherein, the target intent is one of the following: database table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display; Obtain the pre-configured data processing prompt text corresponding to the target intent, and input the prompt text and the target text into a large model; wherein, the large model is obtained by training based on a mixture of sample data from multiple tasks; The large model is scheduled to execute the task with the target intent, and the processing result output by the large model is obtained.

2. The method according to claim 1, characterized in that, The training process of the large model includes: For each intent, a pre-configured data processing prompt text corresponding to that intent is obtained, wherein the data processing prompt text includes the task instruction, task constraints, and business constraints corresponding to the target intent; the data processing prompt text, pre-configured sample data, and corresponding sample processing results are input into the large model; the large model is fine-tuned according to the LoRa fine-tuning algorithm to determine the parameters used by the large model when executing the task corresponding to the intent.

3. The method according to claim 2, characterized in that, If the intent is Text2SQL or SQL2Text, the data processing prompt text also includes database table structure information.

4. The method according to claim 3, characterized in that, The process of determining the database table structure information includes: Based on a preset embedding representation algorithm, the target feature vector corresponding to the target text is determined; Obtain each first feature vector corresponding to the summary description of each saved database table, and determine a first preset number of target database tables that are most similar to the target text based on each first feature vector and the target feature vector; Obtain each second feature vector corresponding to each field description of the saved target database table, and determine a second preset number of target fields that are most similar to the target text based on the second feature vector and the target feature vector; The table names of the first preset number of target database tables and the field information of the second preset number of target fields are determined as the database table structure information.

5. The method according to claim 1, characterized in that, If the target intent is Text2SQL, the processing result of obtaining the output of the large model includes: Obtain the target SQL statement output by the large model; The method further includes: Execute the target SQL statement; If the execution fails, the pre-configured SQL repair prompt text and the target SQL statement will be input into the large model, so that the large model can repair the target SQL statement. Obtain the repaired target SQL statement output by the large model.

6. The method according to claim 5, characterized in that, The method further includes: Execute the repaired target SQL statement; If the execution fails, output a response message indicating that the process failed.

7. The method according to claim 1, characterized in that, The task of scheduling the large model to execute the target intent includes: The large model is scheduled to determine the target parameters used when performing the task with the target intent; The large model is controlled to process the target text according to the target parameters.

8. A data processing device based on a large model, characterized in that, The device includes: The intent recognition module is used to receive target text input by the user and determine the target intent of the target text; wherein, the target intent is one of the following: database table management, Text2SQL, SQL2Text, data generation, table field optimization, and chart display. The processing module is used to obtain the pre-configured data processing prompt text corresponding to the target intent, input the prompt text and the target text into the large model; wherein, the large model is obtained by training based on sample data of multiple tasks; schedule the large model to execute the task of the target intent, and obtain the processing result output by the large model.

9. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to implement the steps of the large model-based data processing method as described in any one of claims 1-7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the large model-based data processing method as described in any one of claims 1-7.

Images