Data processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: determining a target sample corpus based on a sample corpus set and user input information through a similarity estimation model; the target sample corpus represents a sample corpus with the maximum similarity with the user input information in the sample corpus set; splicing the target sample corpus with the user input information to obtain a prompt word text; and through a structured query language (SQL) generation model, based on the cue word text, determining SQL statements related to the user.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In database management, the Text to Structured Query Language (Text-to-SQL) technology can convert human natural language into executable database SQL instructions. Users only need to enter database table information and questions to be queried, and the Text-to-SQL module can directly convert this information into corresponding SQL instructions. Introducing large language model technology into Text-to-SQL tasks has also become the mainstream method in recent years. However, related technical methods still have problems such as high computing cost and low efficiency, and cannot well guide large models to perform Text-to-SQL tasks. Summary of the invention

[0003] To solve the above technical problems, embodiments of the present invention provide a data processing method, apparatus, device, storage medium and program product.

[0004] The data processing method provided in the embodiment of the present application includes:

[0005] Determine a target sample corpus based on the sample corpus set and the user input information through a similarity estimation model; the target sample corpus represents the sample corpus in the sample corpus set that has the greatest similarity to the user input information;

[0006] Concatenate the target sample corpus with the user input information to obtain a prompt word text;

[0007] The structured query language SQL generation model is used to determine the user-related SQL statements based on the prompt word text.

[0008] The data processing device provided in the embodiment of the present application includes:

[0009] An estimation unit, configured to determine a target sample corpus based on a sample corpus set and user input information through a similarity estimation model; the target sample corpus represents a sample corpus in the sample corpus set that has the greatest similarity to the user input information;

[0010] A concatenation unit, used for concatenating the target sample corpus with the user input information to obtain a prompt word text;

[0011] A generating unit is used to determine a user-related SQL statement based on the prompt word text by using a structured query language SQL generation model.

[0012] The processing device provided in the embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any one of the above-mentioned data processing methods.

[0013] The computer-readable storage medium provided in the embodiment of the present application is used to store a computer program, and the computer program enables a computer to execute any one of the above-mentioned data processing methods.

[0014] The computer program product provided in the embodiments of the present application includes computer program instructions, which enable a computer to execute any one of the above-mentioned data processing methods.

[0015] In the technical solution of the embodiment of the present application, the target sample corpus is determined based on the sample corpus set and the user input information through the similarity estimation model, the target sample corpus is spliced ​​with the user input information to obtain the prompt word text, and the SQL statement related to the user is determined based on the prompt word text through the SQL generation model, wherein the target sample corpus represents the sample corpus with the greatest similarity to the user input information in the sample corpus set. In this way, by training a small similarity estimation model, the multi-dimensional information of different questions and database tables is integrated to optimize the similar sample selection process in the context learning prompt word, which can not only adaptively construct prompt words according to user needs, more accurately guide the large model to generate target SQL, but also avoid fine-tuning the parameters of the large language model and reduce the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0017] Figure 2 It is a flowchart of a Text-to-SQL method based on adaptive prompt words provided in an embodiment of the present application;

[0018] Figure 3 It is a schematic diagram of the principle of constructing training samples provided in an embodiment of the present application;

[0019] Figure 4 It is a schematic diagram of the principle of the online process in the Text-to-SQL method based on adaptive prompt words provided in the embodiment of the present application;

[0020] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0021] Figure 6 It is a schematic diagram of the structure of the processing device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0024] In the era of artificial intelligence and big data, the agility of users in obtaining data from databases is crucial. In database management, Text-to-SQL technology can convert human natural language into executable database SQL instructions. Users only need to enter database table information (table name, field name, field type, key value information, etc.) and the questions to be queried, and the Text-to-SQL module can convert these information into corresponding SQL instructions. In addition, the introduction of the Text-to-SQL module in database management can also reduce the workload and learning cost of manually writing SQL instructions, significantly reducing the difficulty of obtaining big data and the threshold for use. In recent years, large language model technology has gradually matured, and its application capabilities in natural language understanding, text generation, logical reasoning, etc. have gradually increased. More and more work has introduced large language models into Text-to-SQL tasks.

[0025] Among them, there are two mainstream methods for using large language models to complete Text-to-SQL tasks: the first method is to collect Text-to-SQL question-answer pair data and fine-tune the large language model. During the fine-tuning process, information such as table structure and user questions is used as the input of the large language model, and the corresponding SQL statement is used as the target output of the large language model. However, due to the large number of parameters in the large language model, the computational and time costs required for fine-tuning the large language model are also high; the second method is to use contextual learning to design prompts containing multiple Text-to-SQL examples and instructions for standardizing the output of the large model to guide the large language model to complete the SQL generation task. This method does not require changing the parameters of the large language model. The relevant technical solutions mainly include the following:

[0026] (1) Automatic question answering method based on semi-supervised learning and Text-to-SQL model: First, a large number of samples are automatically generated for model training based on manually annotated {question, SQL} samples, and then the Sequence-to-Slot model architecture is used to complete the SQL generation task based on the trained model. The Sequence-to-Slot model architecture includes several different sub-modules for predicting SQL sub-fragments, and an entity category encoding module for identifying the category to which the entity in the question belongs in the database. The entity category encoding module is used to take the entity category in the question as the input of the sub-module to encode the entity category.

[0027] (2) Two-stage Text-to-SQL generation method based on large language model: In the first stage, the required tables and columns are selected in the corresponding database according to the input question; in the second stage, the large model is used to generate SQL statements based on the tables and columns selected in the first stage. In the first stage, a small model is trained to select tables and columns, and in the second stage, the large language model is directly used to complete the SQL statement generation.

[0028] (3) Adaptive rule-guided large language model SQL generation method: First, standardize the library table information, merge the table name, column name and column data type into dictionary type data, and normalize the format; then, based on the context understanding ability of the large language model, determine whether the filtered column conditions need to cross tables, and combine the table structure knowledge to infer the SQL command for merging tables; then merge the intermediate SQL fragments obtained by the table name and column name filtering module, column condition extraction module, and table merging module into a complete SQL query statement, and perform syntax and consistency checks on the generated SQL query statement.

[0029] (4) SQL generation method for a large language model based on prompt word optimization: First, the domain-related entity information in the user question and sample question is masked to obtain user text and sample text in question format only; then, these texts are represented by vectors, and the similarity between each sample text and the user text is calculated, so as to select samples that are similar to the user question and use them to concatenate prompt words to better stimulate the SQL generation capability of the large language model.

[0030] However, the above solution still has the following problems:

[0031] (1) The first solution above collects data through sample augmentation and then uses this data to train a proprietary model. However, the quality of sample augmentation depends heavily on the performance of the generation unit. If the performance of the generation unit is poor, it is difficult to obtain high-quality training data. Moreover, the diversity of the generated data is limited by manually annotated data. At the same time, end-to-end training of a proprietary SQL generation model cannot fully utilize the knowledge that the large model has learned, and its training process requires a huge amount of data and computing power costs.

[0032] (2) The second solution above decomposes the Text-to-SQL task into two stages. First, the table and field need to be screened according to the user input question. In this step, a proprietary model still needs to be trained from scratch, and a large number of samples need to be collected. If the table and field screening are wrong, the error will be passed to the next SQL generation result, resulting in weak overall robustness. In addition, during the SQL generation process of the large model, this solution does not optimize the input prompt words, and cannot better guide the large model to perform the Text-to-SQL task.

[0033] (3) The third solution mentioned above requires splitting the different steps of Text-to-SQL, designing different prompt words for each step, and calling the large language model multiple times. In addition, this solution requires human intervention in the SQL generation process, because manual configuration is required to build prompt words for multiple steps, which will affect the overall task deployment efficiency.

[0034] (4) The fourth solution mentioned above is to optimize the input prompt words of the large language model by determining similar examples through question similarity in context learning. However, in addition to being related to questions, SQL statements are also closely related to table structures. Similar questions may have different corresponding SQL statements under different table structures. Therefore, selecting similar examples only by question similarity will lose the table structure information. If the selected examples are very different from the correct SQL structure, it will mislead the large language model to generate incorrect SQL.

[0035] In order to solve the above technical problems, this application proposes a data processing method for adaptive prompt word construction for Text-to-SQL tasks, constructs training data through comparative learning, and trains a small similarity estimation model, thereby optimizing the similar sample selection process in the context learning prompt words, and guiding the large model to better generate the target SQL. Among them, in order to solve the problem of high computing cost caused by fine-tuning the large model, this solution adopts context learning to train a small similarity estimation model to optimize the input prompt words of the large model to better guide the large model to output SQL, thereby avoiding fine-tuning the parameters of the large language model and reducing the computing cost. In order to solve the problem of inaccurate sample selection caused by screening similar samples only based on the similarity of the problem, this solution integrates the multi-dimensional information of different problems and database tables by training a small similarity estimation model, avoiding the problem that similar problems correspond to different SQLs and misleading the large model to generate wrong SQL, and can adaptively construct prompt words according to user needs, and more accurately guide the large model to generate target SQL.

[0036] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0037] The present application embodiment proposes a data processing method. Figure 1 is a flow chart of a data processing method provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0038] Step 101: Determine a target sample corpus based on a sample corpus set and user input information through a similarity estimation model.

[0039] In the embodiment of the present application, in order to obtain the sample in the corpus that is most similar to the user input information, the most similar sample (target sample corpus) can be determined based on the sample corpus set and the user input information through the similarity estimation model, that is, the sample corpus set and the user input information are input into the similarity estimation model, so that the similarity estimation model outputs the target sample corpus according to the similarity between each sample in the sample corpus set and the user input information. Among them, the target sample corpus represents the sample corpus in the sample corpus set that has the greatest similarity with the user input information.

[0040] Here, before determining the target sample corpus based on the sample corpus set and user input information through the similarity estimation model, the sample corpus set and user input information need to be preprocessed. For the preprocessing of the sample corpus set, it is necessary to correct the irregular column names and organize the database structure information into the language format of SQL to create tables, etc.; for the preprocessing of user input information, it includes table coarse screening and field filtering. Table coarse screening is to screen the database structure information input by the user based on the user question, and only select the table that is most likely to be used by the user question. Field filtering is to compare the user question with each field in the database structure information, and only retain the field with the strongest relevance to the user question. Each sample corpus in the preprocessed sample corpus set includes three elements: sample database structure information, sample question and correct SQL statement. The preprocessed user input information includes database structure information and user question.

[0041] It should be noted that the source of the sample corpus collection can be an authoritative public data set in the field of Text-to-SQL, such as public data sets on websites such as Spider and BIRD. The samples in these public data sets contain database structure information, questions, and corresponding SQL. Alternatively, you can also build a sample corpus collection by writing relevant programs to crawl data on specific topics from the Internet. Here, as long as the source of the sample corpus collection is legal and compliant, there is no restriction on its collection method.

[0042] Based on this, in some implementations, step 101 can be implemented through steps 1011-1013, which specifically include:

[0043] Step 1011: each sample corpus in the sample corpus set is sequentially concatenated with the user input information to obtain multiple text blocks.

[0044] In the embodiment of the present application, in order to determine the target sample corpus based on the sample corpus set and the user input information through the similarity estimation model, each sample corpus in the sample corpus set can be sequentially concatenated with the user input information to obtain multiple text blocks. For example, if each sample corpus in the sample corpus set can be represented as are respectively the sample database structure information, sample question and correct SQL statement of the i-th sample in the sample corpus set; the user input information can be expressed as u=(u s ,u q ),u s ,u q are the database structure information and user questions input by the user respectively, then the text block of each sample corpus concatenated with the user input information can be expressed as Among them, '|' is a text concatenation operation.

[0045] Step 1012: Input multiple text blocks into a similarity estimation model, and output multiple similarity ratings through the similarity estimation model.

[0046] In an embodiment of the present application, after obtaining multiple text blocks spliced ​​by each sample corpus in the sample corpus set and the user input information, the multiple text blocks are input into a similarity estimation model, so that the similarity estimation model performs a similarity score on each text block to obtain multiple similarity scores, and outputs multiple similarity ratings after the multiple similarity scores are discretized and graded.

[0047] Based on this, in some implementations, step 1012 may specifically include:

[0048] Inputting multiple text blocks into a similarity estimation model, calculating the similarity score between each sample corpus and the user input information through the similarity estimation model, and obtaining multiple similarity scores;

[0049] The multiple similarity scores are divided into levels to obtain multiple similarity ratings, and the multiple similarity ratings are output through a similarity estimation model.

[0050] Here, after multiple text blocks are input into the similarity estimation model, the similarity estimation model will calculate the similarity score between the corresponding sample corpus in each text block and the user input information, thereby obtaining multiple similarity scores. Since only the most similar sample corpus needs to be determined, the specific similarity gap value between each sample corpus and the user input information is not important. Therefore, multiple similarity scores can be divided into discrete levels to reflect the degree of similarity. For example, the similarity rating can be divided into four levels: 1, 2, 3, and 4, which represent the four results of "very similar", "similar", "different", and "very different" in order. Assuming that the upper limit of the similarity score is 100 and the lower limit is 0, if the sample corpus corresponding to the text block and the user input information are very similar, then the similarity score is 0. If the similarity score of the sample corpus in the text block is in the range of 80-100, the corresponding similarity rating is 1, indicating that the sample corpus is very similar to the user input information; if the similarity score between the sample corpus in the text block and the user input information is in the range of 60-80, the corresponding similarity rating is 2, indicating that the sample corpus is similar to the user input information; if the similarity score between the sample corpus in the text block and the user input information is in the range of 40-60, the corresponding similarity rating is 3, indicating that the sample corpus is different from the user input information; if the similarity score between the sample corpus in the text block and the user input information is in the range of 0-40, the corresponding similarity rating is 4, indicating that the sample corpus is very different from the user input information.

[0051] Step 1013: Determine a target sample corpus based on multiple similarity ratings.

[0052] In an embodiment of the present application, after obtaining the similarity rating between each sample corpus in the sample corpus set and the user input information, a negative correlation can be set between the similarity rating and the similarity score, so that the sample corpus with the largest similarity score with the user input information has the smallest corresponding similarity rating. Therefore, the smallest similarity rating can be selected from multiple similarity ratings, and the sample corpus corresponding to the smallest similarity rating in the sample corpus set is determined as the target sample corpus. For example, if there are five similarity rating results of Class = (1, 2, 3, 4, 5), the sample corpus corresponding to the similarity rating of 1 has the largest similarity with the user input information, and the sample corpus can be used as the target sample corpus.

[0053] Based on this, in some implementations, step 1013 may specifically include: selecting the smallest similarity rating from multiple similarity ratings, and using the sample corpus corresponding to the smallest similarity rating in the sample corpus set as the target sample corpus. The similarity rating is negatively correlated with the similarity score.

[0054] Step 102: concatenate the target sample corpus with the user input information to obtain the prompt word text.

[0055] In the embodiment of the present application, after determining the target sample corpus, the target sample corpus and the user input information can be spliced ​​to obtain the input of the SQL generation model, that is, the prompt word text. Among them, since the prompt word text is composed of the target sample corpus and the user input information, and the similarity between the target sample corpus and the user input information is the greatest, the prompt word text contains the sample database structure information, sample questions, correct SQL statements of the target sample corpus, and the database structure information of the user input information and the user question.

[0056] Here, in the process of generating the prompt word text, the target sample corpus is first analyzed, that is, the sample database structure information including the table name, field name and its data type is extracted, and the sample question is understood, its key information and query intent are identified, and the correct SQL statement is analyzed to understand its structure, the functions and conditions used, etc.; then the user input information is analyzed, that is, the database structure information of the user input information is extracted, and it is compared with the sample database structure information to identify the differences and commonalities, and the user question is understood, its key information and query intent are identified, and it is compared with the sample question to identify the differences and commonalities; finally, according to the analysis results, the task instructions are written to clarify the tasks that the SQL generation model needs to perform, that is, to generate the target SQL statement corresponding to the database structure information and user question input by the user; thus, the prompt word text finally generated includes the task instructions, the target database structure (the table name and field name involved in the user question, etc.), the target question (user question), the sample question and the corresponding correct SQL statement.

[0057] Step 103: Determine the user-related SQL statement based on the prompt word text through the SQL generation model.

[0058] In the embodiment of the present application, in order to further guide the SQL generation model to complete the SQL generation task, after obtaining the prompt word text, the prompt word text can be used as the input of the SQL generation model and input into the SQL generation model, so that the SQL generation model outputs the target SQL statement according to the prompt word text. Specifically, since the prompt word text includes task instructions, target database structure, target problem, sample problem and corresponding correct SQL statement, the SQL generation model will generate the target SQL statement corresponding to the target database structure information and the target problem according to the task instructions, combined with the sample problem and the correct SQL statement. Among them, the SQL generation model is a large language model with SQL generation capability.

[0059] Here, the big language model is a deep learning model trained with a large amount of text data, designed to understand and generate human language. The big language model is based on a deep neural network architecture. By training a large-scale corpus, it can capture the semantic and grammatical rules between words, phrases and sentences, thereby generating coherent and natural text. Among them, the big language model has the ability to handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and the big language model used to generate SQL has the ability to convert natural language questions into SQL query statements.

[0060] Based on this, in some implementations, step 103 may specifically include:

[0061] Inputting the prompt word text into the SQL generation model, so that the SQL generation model generates a target SQL statement corresponding to the target database structure and the target question according to the task instruction, the sample question and the correct SQL statement, and outputs the target SQL statement;

[0062] Verify and modify the target SQL statement to obtain the user-related SQL statement.

[0063] Here, after the SQL generation model outputs the target SQL statement, the target SQL statement can also be verified and corrected, such as checking whether the column name is correct, whether the column name comes from the corresponding table, etc., so that after verifying and correcting the target SQL statement, the final user-related SQL statement is obtained.

[0064] Based on the foregoing embodiment, in some implementations, during the training process of the similarity estimation model, the sample corpus set may be divided into a training corpus set and a verification corpus set. The training process of the similarity estimation model may be implemented through steps S1-S5, specifically including:

[0065] Step S1: Select a first sample corpus and a second sample corpus from a training corpus set.

[0066] Step S2: Determine training samples based on the first sample corpus and the second sample corpus.

[0067] In the embodiment of the present application, for the construction of training samples, two samples may be randomly selected from the training corpus set, namely, a first sample corpus and a second sample corpus, and the training samples are constructed based on the first sample corpus and the second sample corpus. The first sample corpus includes a first database structure, a first question, and a first SQL statement, and the second sample corpus includes a second database structure, a second question, and a second SQL statement.

[0068] Based on this, in some implementations, step S2 can be implemented through steps S21-S24, specifically including:

[0069] Step S21: concatenate the first database structure, the first question and the first SQL statement with the second database structure and the second question to obtain a training text block.

[0070] Here, by concatenating the first database structure, the first question, and the first SQL statement in the first sample corpus with the second database structure and the second question in the second sample corpus, a training text block is obtained, which can be used as the data feature part in the training sample. For example, the first sample corpus can be represented as are respectively the sample database structure information, sample question and correct SQL statement of the i-th sample in the training corpus set; the second sample corpus can be expressed as are the sample database structure information, sample question and correct SQL statement of the jth sample in the training corpus set respectively. Then the training text block obtained by concatenating the first sample corpus with the second sample corpus can be expressed as Among them, '|' is a text concatenation operation.

[0071] Step S22: Calculate the similarity score between the first sample corpus and the second sample corpus.

[0072] Here, after randomly selecting the first sample corpus and the second sample corpus from the training corpus set, the similarity score between the first sample corpus and the second sample corpus can be calculated according to the similarity score calculation function. The similarity score between the first sample corpus and the second sample corpus refers to the similarity score between the first sample corpus and the second sample corpus for the SQL statement.

[0073] Based on this, in some implementations, step S22 may specifically include:

[0074] Determine an SQL keyword set, and according to the SQL keyword set, count the keywords appearing in the first SQL statement and the second SQL statement respectively to obtain a first keyword set and a second keyword set;

[0075] Count the number of occurrences of each keyword in the first keyword set in the first SQL statement, and the number of occurrences of each keyword in the second keyword set in the second SQL statement, to obtain a first number code and a second number code;

[0076] Based on the first number encoding and the second number encoding, a similarity score between the first sample corpus and the second sample corpus is determined.

[0077] Here, you can first organize and determine the commonly used SQL keyword set, and then, based on the keywords in the SQL keyword set, count the keywords that appear in the first SQL statement and the keywords that appear in the second SQL statement, so as to obtain a first keyword set and a second keyword set, wherein the first keyword set records all the keywords that appear in the SQL keyword set in the first SQL statement, and the second keyword set records all the keywords that appear in the SQL keyword set in the second SQL statement.

[0078] After obtaining the first keyword set and the second keyword set, the number of occurrences of each keyword in the first keyword set in the first SQL statement and the number of occurrences of each keyword in the second keyword set in the second SQL statement are counted respectively to obtain a first number code and a second number code, wherein the first number code records all keywords in the SQL keyword set and the corresponding keyword times in the first SQL statement, and the second number code records all keywords in the SQL keyword set and the corresponding keyword times in the second SQL statement, and both the first number code and the second number code are expressed in vector form.

[0079] After obtaining the first number code and the second number code, the distance between the first number code and the second number code can be calculated, which can be L1 distance or L2 distance, etc., and the negative of the distance is used as the similarity score between the first sample corpus and the second sample corpus.

[0080] For example, the SQL keyword set is represented as W = {w1,w2,…,w w}, according to the SQL keyword set, count the SQL statements of the first sample corpus and the second sample corpus respectively and The keywords and the number of times the keywords appear in the code are extracted, and the first number of keywords is encoded. and the second order code Calculate two SQL statements based on the first number code and the second number code and The L1 distance corresponding to the keyword frequency encoding in , and the negative of the L1 distance is used as the first sample corpus s i and the second sample corpus s j The similarity score between

[0081] Based on this, in some embodiments, determining the similarity score between the first sample corpus and the second sample corpus based on the first number coding and the second number coding may specifically include: calculating the distance between the first number coding and the second number coding, and taking the negative of the distance as the similarity score between the first sample corpus and the second sample corpus.

[0082] Based on this, in some implementations, step S22 may further specifically include:

[0083] Determine the frequency of occurrence of the corresponding keyword in the SQL keyword set;

[0084] Based on historical knowledge and keyword occurrence frequency, determine the weight value of each keyword in the SQL keyword set in the distance calculation;

[0085] According to the weight value of each keyword in the distance calculation, the weighted distance between the first digital code and the second digital code is calculated, and the negative of the weighted distance is used as the similarity score between the first sample corpus and the second sample corpus.

[0086] Here, while determining the commonly used SQL keyword set as a whole, the frequency of occurrence of each keyword in the SQL keyword set can also be determined, and then the weight value of each keyword in the SQL keyword set in the distance calculation is defined based on historical knowledge (expert knowledge) and the frequency of occurrence of each keyword, and the keyword weight is set to be negatively correlated with the keyword occurrence frequency to enhance the importance of rare keywords, that is, the lower the frequency of occurrence of the keyword, the greater its weight value.

[0087] After determining the weight value of each keyword in the SQL keyword set in the distance calculation, the weighted distance between the first digital code and the second digital code can be calculated according to the weight value of each keyword in the distance calculation. The weighted distance can be a weighted L1 distance or an L2 weighted distance, etc., and the negative of the weighted distance is used as the similarity score between the first sample corpus and the second sample corpus.

[0088] For example, the SQL keyword set is represented as W = {w1,w2,…,w w}, the frequency of occurrence of each keyword in the keyword set is expressed as First, according to expert knowledge and keyword frequency, the weight value of each keyword in the distance calculation is defined as η = {η1, η2, …, η w Then, according to the SQL keyword set, the SQL statements of the first sample corpus and the second sample corpus are counted respectively. and The keywords and the number of times the keywords appear in the code are extracted, and the first number of keywords is encoded. and the second order code Finally, based on the first number code and the second number code, combined with the weight value of each keyword in the distance calculation, two SQL statements are calculated. and The weighted L1 distance corresponding to the keyword frequency encoding in , and the negative of the weighted L1 distance is used as the first sample corpus s i and the second sample corpus s j The similarity score between

[0089] Step S23: Classify the similarity scores between the first sample corpus and the second sample corpus according to levels to obtain a similarity rating between the first sample corpus and the second sample corpus.

[0090] Here, after calculating the similarity score between the first sample corpus and the second sample corpus, the similarity score may be discretely divided according to levels to obtain corresponding similarity ratings. For example, the similarity rating can be set to 1, 2, 3, and 4, which represent the four results of "very similar", "similar", "different", and "very different" in order. Assuming that the upper limit of the similarity score is 100 and the lower limit is 0, if the similarity score between the first sample corpus and the second sample corpus is in the range of 80-100, then the corresponding similarity rating is 1, indicating that the first sample corpus and the second sample corpus are very similar; if the similarity score between the first sample corpus and the second sample corpus is in the range of 60-80, then the corresponding similarity rating is 2, indicating that the first sample corpus and the second sample corpus are similar; if the similarity score between the first sample corpus and the second sample corpus is in the range of 40-60, then the corresponding similarity rating is 3, indicating that the first sample corpus and the second sample corpus are different; if the similarity score between the first sample corpus and the second sample corpus is in the range of 0-40, then the corresponding similarity rating is 4, indicating that the first sample corpus and the second sample corpus are very different.

[0091] Step S24: Determine a training sample based on the similarity ratings between the training text block and the first sample corpus and the second sample corpus.

[0092] Here, after obtaining the training text blocks and similarity ratings of the first sample corpus and the second sample corpus respectively, the training sample block can be used as the data feature part, and the similarity rating can be used as the data label part. The training sample block and the similarity rating are spliced ​​to obtain training samples for the first sample corpus and the second sample corpus.

[0093] Step S3: Select a third sample corpus from the training corpus set, and select a fourth sample corpus from the verification corpus set.

[0094] Step S4: Determine a verification sample based on the third sample corpus and the fourth sample corpus.

[0095] In an embodiment of the present application, for the construction of a verification sample, a sample can be randomly selected from the training corpus set as the third sample corpus, and a sample can be randomly selected from the verification corpus set as the fourth sample corpus, and the verification sample can be constructed based on the third sample corpus and the fourth sample corpus.

[0096] It should be noted that the construction process of the verification sample is similar to that of the training sample. The specific process can refer to the construction content of the training sample mentioned above, and will not be elaborated here.

[0097] Step S5: Train the model using training samples, and adjust the model using verification samples during the training process to obtain a similarity estimation model.

[0098] In the embodiment of the present application, after constructing the training samples and the verification samples, the similarity estimation model can be trained based on the training samples, with the training text blocks in each training sample as the model input and the corresponding similarity rating as the target output. At the same time, during the training process of the similarity estimation model, the model is preliminarily evaluated using the verification samples, the performance of the model on the verification samples is observed, and the model hyperparameters are adjusted according to the performance, such as the number of neural network layers, the number of neurons in each layer, and the regularization parameters, so as to train a similarity estimation model with strong generalization ability and prevent the model from overfitting.

[0099] It should be noted that after the similarity estimation model is obtained through training based on the training samples and the validation samples, a test sample can also be constructed based on the sample corpus set, so that the test sample can be used to evaluate the generalization ability of the trained similarity estimation model and verify its performance on unknown data. In the construction of training samples, since the purpose of the test sample is to evaluate the performance of the model on unknown data, it is only necessary to construct the input data without constructing the output label.

[0100] In the technical solution of the embodiment of the present application, the target sample corpus is determined based on the sample corpus set and the user input information through the similarity estimation model, the target sample corpus is spliced ​​with the user input information to obtain the prompt word text, and the SQL statement related to the user is determined based on the prompt word text through the SQL generation model, wherein the target sample corpus represents the sample corpus with the greatest similarity to the user input information in the sample corpus set. In this way, by training a small similarity estimation model, the multi-dimensional information of different questions and database tables is integrated to optimize the similar sample selection process in the context learning prompt word, which can not only adaptively construct prompt words according to user needs, more accurately guide the large model to generate target SQL, but also avoid fine-tuning the parameters of the large language model and reduce the computational cost.

[0101] The present application also proposes a Text-to-SQL method based on adaptive prompt words. Figure 2 is a flowchart of a Text-to-SQL method based on adaptive prompt words provided in an embodiment of the present application, such as Figure 2 As shown in the figure, the method flow includes two parts: offline process and online process. The offline process includes: (1) early sample corpus collection and preprocessing for subsequent model training and sample selection; (2) using sample corpus to construct training and verification data sets; (3) using training and verification data sets to train similarity estimation small model. The online process includes: (1) obtaining user input information, including user library table information and user questions, and normalizing the user input information; (2) splicing user input information and multiple samples in the sample corpus into multiple inputs of the small model; (3) inputting multiple splicing results into the similarity estimation small model for similarity evaluation; (4) splicing user input information and the most similar sample selected according to the similarity evaluation into input prompt words; (5) the large model outputs SQL statements according to the input prompt words, and the verification module verifies and corrects the generated SQL statements.

[0102] The offline process and the online process specifically include the following steps:

[0103] Offline process

[0104] Step 201: Preprocess the sample corpus.

[0105] There are a large number of authoritative public data sets in the field of Text-to-SQL, including Spider, BIRD, etc. The samples in these public data sets contain database table information, questions, and corresponding SQL. These samples are aggregated as a sample corpus.

[0106] After obtaining the sample corpus, the corpus in the sample corpus is preprocessed, including correcting irregular table names and column names, organizing database table information into SQL table creation language format, etc. Each preprocessed sample corpus includes three elements: database table schema information, sample questions, and corresponding correct SQL statements.

[0107] Step 202: Generate training samples and verification samples for a similarity estimation model based on the sample corpus.

[0108] During the training process of the similarity estimation model, all sample corpora are divided into two sets: set S simulates the corpus composed of public data sets and collected corpora in actual applications, and set A corpus of examples that simulate questions asked by users but do not know the correct SQL. Specifically:

[0109] The i-th example s in the set S i ∈S and in They are the schema information, problem, and correct SQL of the i-th sample respectively.

[0110] gather The i-th example in and in They are the schema information, problem, and correct SQL of the i-th sample respectively.

[0111] In order to train the similarity estimation model, it is necessary to construct training samples and validation samples for the similarity estimation model. The overall idea of ​​model training is: for two samples, one sample knows the database schema information, the problem and the corresponding correct SQL, and the other sample knows the database schema information and the problem, but does not know the corresponding correct SQL. The similarity estimation model needs to compare the SQL similarity of the two samples without knowing the correct SQL of the second sample. Because the second sample does not clearly give the SQL, it is necessary to train the similarity estimation model to learn how to estimate SQL similarity based on known information.

[0112] First, randomly select two samples s from the set S i and j , this step is to construct training samples for the similarity estimation model, so both samples are selected from the set S.

[0113] Secondly, the sample s i Schema information, questions, SQL and examples j The schema information and questions are concatenated into a text block. Inst is the instruction for guiding the task, and '|' is the text concatenation operation.

[0114] Then, according to the sample s i and examples j Calculate the similarity score between two samples The similarity score is calculated based on the SQL statements of the two examples, and the similarity estimation model evaluates the SQL similarity of the two examples. When generating training data, it is known that the second example s j The second example is from a pre-collected corpus, so the similarity estimation model can be guided to learn how to estimate SQL similarity based on known information by calculating the similarity score. The similarity score calculation function can be defined by the user. Two implementation methods are given below:

[0115] Method 1: Organize the commonly used SQL keyword set W = {w1,w2,…,w w}. Based on the SQL statements of the two examples and Extract the keywords in the code to see the number of occurrences of the keywords and Calculate the L1 distance between the number of keyword occurrences in two SQL statements, and use the negative of the L1 distance as sample s i and examples j The similarity score is calculated as:

[0116]

[0117] Method 2: Organize the commonly used SQL keyword set W = {w1,w2,…,w w} and the frequency of occurrence of each keyword in the keyword set According to the two sample SQL statements and Extract the keywords in the code to see the number of occurrences of the keywords and According to expert knowledge and keyword frequency, the weight of each keyword in the distance calculation is defined as η = {η1, η2, …, η w}. Among them, the keyword weight and keyword frequency show a negative correlation to enhance the importance of rare keywords. Calculate the weighted L1 distance of the number of keyword occurrences in two SQL statements, and use the negative of the weighted L1 distance as the sample s i and examples j The similarity score is calculated as:

[0118]

[0119] Finally, after calculating the sample s i and examples j After the similarity score sc is obtained, the similarity score sc is divided into discrete levels. For example, the similarity level can be divided into Class m ∈{1,2,3,4}, where {1,2,3,4} represent the four levels of results: “very similar”, “similar”, “different”, and “very different”. m is the data feature, Class m is the data label, and the training sample is (Input m ,Class m ).

[0120] like Figure 3 The figure shows a schematic diagram of the principle of constructing training samples provided by an embodiment of the present application, wherein the input is the concatenation of the schema information, question and corresponding SQL of sample 1 and the schema information and question of sample 2. The output target label is the result of discretization and grading of the similarity scores of the two sample SQLs.

[0121] The construction process of the validation sample is similar to that of the test sample.

[0122] First, randomly select a sample s from the set S i , from the collection Randomly select a sample from Among them, in the process of building the verification sample, the second sample needs to be from the collection Select in to ensure that the training set samples do not appear in the test set.

[0123] Secondly, the sample s i Schema information, questions, SQL and examples The schema information and questions are concatenated into a text block. Inst is the instruction for guiding the task, and '|' is the text concatenation operation.

[0124] Then, according to the sample s i and examples information, calculate the similarity score between the two samples sc1 =

[0125]

[0126] Finally, the similarity score sc1 is divided into Class n ∈{1,2,3,4}. In this way, the final Input nis the data feature, Class n is the data label, and the verification sample is (Input n ,Class n ).

[0127] Step 203: training a similarity estimation model.

[0128] After constructing the training sample and the validation sample, based on the training sample, m is the input of the model, Class m The label is the target output of the model to train the similarity estimation model. Among them, you can choose to generate a model in the model selection, and then output Class m The labels are converted to character form. For example, if there are four levels, the model output is {'1', '2', '3', '4'}; or you can choose a classification model, then the output dimension of the classification model is 4, each dimension corresponds to the rating level.

[0129] During the training process of the similarity estimation model, the validation samples are used to perform a preliminary evaluation of the model, observe the performance of the model on the validation samples, and adjust the model hyperparameters based on these performances, such as the number of neural network layers, the number of neurons in each layer, and the regularization parameters. Using validation samples to adjust the model helps select a model with strong generalization ability and prevent overfitting.

[0130] Online process

[0131] like Figure 4 The figure shows a schematic diagram of the principle of the online process in the Text-to-SQL method based on adaptive prompt words provided in the embodiment of the present application. First, the user input information is received and preprocessed, and then the sample and the user input information are spliced ​​as the input of the similarity estimation model. Finally, according to the evaluation result of the similarity estimation model, the most similar sample is selected and an adaptive few-shot prompt word is constructed, and then it is handed over to the large model to complete the SQL generation task.

[0132] Step 204: Pre-process the user input information.

[0133] After obtaining the trained similarity estimation model, in the online process, the second sample input to the similarity estimation model becomes the schema information and question input by the user. The number of tables and fields in the tables input by the user may be large, and both the similarity estimation model and the large language model that generates SQL have strict requirements on the input length. The fewer characters in the text, the better. In fact, the difference in user questions will only cause differences in the tables used and the fields involved, and will not involve all tables and fields. Therefore, irrelevant tables and fields can be filtered out according to user questions. Two steps can be added in this step to shorten the large model input: table coarse screening and field filtering. Table coarse screening is based on user questions, filtering the schema information input by the user, and only selecting the tables that are most likely to be used by the user questions. Field filtering is to compare the user question with each field in the schema information, and only retain the fields that are most relevant to the user question.

[0134] Step 205: concatenate the sample with the user input information to obtain a concatenated text block.

[0135] In order to obtain the few-shot prompt word, it is necessary to select similar samples based on the user input information. Each sample in the set S is sequentially concatenated with the user input information to construct the input results of N similarity estimation models. The input format is

[0136] Step 206: Use a similarity estimation model to perform similarity evaluation on the concatenated text blocks, and select the most similar examples therefrom.

[0137] The constructed N samples and the input results of the user input information are input into the similarity estimation model, and the similarity estimation model outputs N similarity rating results. Figure 3 As shown in FIG, if the output of the similarity estimation model is {′1′,′2′,′3′,′4′}, the sample corresponding to ′1′ in {′1′,′2′,′3′,′4′} is selected as the most similar sample.

[0138] Step 207: The most similar example is concatenated with the user input information to construct an adaptive prompt word.

[0139] The most similar examples selected are concatenated with the user input information to construct an adaptive few-shot prompt word.

[0140] Step 208: Generate SQL according to the adaptive prompt word, and check and correct the SQL.

[0141] The constructed adaptive few-shot prompt words are input into the large language model, and the SQL statement is output through the large language model. In the post-verification module, the generated SQL statement is verified and corrected, for example, whether the column name is correct and whether the column name comes from the corresponding table.

[0142] In the technical solution provided in the embodiment of the present application, an adaptive prompt word construction method for Text-to-SQL tasks is proposed. By training a small similarity estimation model, corresponding similar samples can be selected from the sample corpus for different questions, and prompt words can be adaptively constructed according to user needs, thereby more accurately stimulating the large model to complete the SQL generation task. Among them, this solution has the following advantages:

[0143] (1) In terms of computing power cost, compared with the high computing cost caused by the need to train or fine-tune a large model, this solution only needs to train a small similarity estimation model and call the large model to generate SQL in a context learning manner without changing the parameters of the large model. By optimizing the few-sample prompt words required for context learning, without changing the parameters of the large model, the SQL generation knowledge learned by the large model can be fully stimulated to accurately generate the target SQL.

[0144] (2) In terms of sample collection cost, compared with the problem of training dedicated SQL to generate a small model and fine-tuning a large model, which requires collecting a large number of SQL question-answer pairs, this solution uses contrastive learning to construct training samples when training a small similarity estimation model. In theory, n samples can be used to construct n 2 Small model training samples, thus reducing the cost of data collection.

[0145] (3) In terms of the accuracy of constructing prompt words, compared with using only question similarity as an indicator for evaluating sample similarity, this solution integrates multi-dimensional information within the model by training a small model, and various types of information (question similarity, database structure similarity, etc.) jointly affect the similarity estimation between samples. Therefore, the accuracy of similar samples selected by this solution and the robustness of the constructed prompt words are both higher.

[0146] The present application also provides a data processing device. Figure 5 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0147] The estimation unit 501 is used to determine the target sample corpus based on the sample corpus set and the user input information through a similarity estimation model; the target sample corpus represents the sample corpus in the sample corpus set that has the greatest similarity with the user input information.

[0148] The concatenation unit 502 is used to concatenate the target sample corpus with the user input information to obtain the prompt word text.

[0149] The generating unit 503 is used to determine the SQL statement related to the user based on the prompt word text by using the structured query language SQL generating model.

[0150] In some embodiments, the estimation unit 501 is specifically used to sequentially concatenate each sample corpus in the sample corpus set with the user input information to obtain multiple text blocks; input the multiple text blocks into the similarity estimation model, and output multiple similarity ratings through the similarity estimation model; and determine the target sample corpus based on the multiple similarity ratings.

[0151] In some embodiments, the estimation unit 501 is further specifically used to input multiple text blocks into a similarity estimation model, calculate the similarity score between each sample corpus and the user input information through the similarity estimation model, and obtain multiple similarity scores; divide the multiple similarity scores into levels to obtain multiple similarity ratings, and output multiple similarity ratings through the similarity estimation model.

[0152] In some implementations, the estimation unit 501 is further specifically configured to select the smallest similarity rating from multiple similarity ratings, and use the sample corpus corresponding to the smallest similarity rating in the sample corpus set as the target sample corpus; wherein the similarity rating is negatively correlated with the similarity score.

[0153] In some embodiments, the device further comprises:

[0154] A determination unit is used to select a first sample corpus and a second sample corpus from a training corpus set; determine a training sample based on the first sample corpus and the second sample corpus; select a third sample corpus from the training corpus set, and select a fourth sample corpus from a verification corpus set; determine a verification sample based on the third sample corpus and the fourth sample corpus; wherein the sample corpus set includes a training corpus set and a verification corpus set.

[0155] The training unit is used to train the model through training samples and adjust the model through verification samples during the training process to obtain a similarity estimation model.

[0156] In some embodiments, the determination unit is specifically used to concatenate the first database structure, the first question, and the first SQL statement with the second database structure and the second question to obtain a training text block; calculate the similarity score between the first sample corpus and the second sample corpus; divide the similarity score between the first sample corpus and the second sample corpus into levels to obtain a similarity rating between the first sample corpus and the second sample corpus; determine the training sample based on the training text block and the similarity rating between the first sample corpus and the second sample corpus; wherein the first sample corpus includes the first database structure, the first question, and the first SQL statement, and the second sample corpus includes the second database structure, the second question, and the second SQL statement.

[0157] In some embodiments, the determination unit is further specifically used to determine an SQL keyword set, and based on the SQL keyword set, respectively count the keywords appearing in the first SQL statement and the second SQL statement to obtain a first keyword set and a second keyword set; respectively count the number of occurrences of each keyword in the first keyword set in the first SQL statement, and the number of occurrences of each keyword in the second keyword set in the second SQL statement to obtain a first number code and a second number code; based on the first number code and the second number code, determine a similarity score between the first sample corpus and the second sample corpus.

[0158] In some implementations, the determination unit is further specifically configured to calculate a distance between the first digital encoding and the second digital encoding, and use the negative of the distance as a similarity score between the first sample corpus and the second sample corpus.

[0159] In some embodiments, the determination unit is further specifically used to determine the frequency of occurrence of corresponding keywords in the SQL keyword set; based on historical knowledge and the frequency of occurrence of keywords, determine the weight value of each keyword in the SQL keyword set in the distance calculation; according to the weight value of each keyword in the distance calculation, calculate the weighted distance between the first digital code and the second digital code, and use the negative of the weighted distance as the similarity score between the first sample corpus and the second sample corpus.

[0160] In some embodiments, the generation unit 503 is specifically used to input the prompt word text into the SQL generation model, so that the SQL generation model generates a target SQL statement corresponding to the target database structure information and the target problem according to the task instruction, combined with the sample problem and the correct SQL statement, and outputs the target SQL statement; verifies and corrects the target SQL statement to obtain the user-related SQL statement; wherein the prompt word text includes the task instruction, the target database structure information, the target problem, the sample problem and the corresponding correct SQL statement.

[0161] In the technical solution of the embodiment of the present application, the target sample corpus is determined based on the sample corpus set and the user input information through the similarity estimation model, the target sample corpus is spliced ​​with the user input information to obtain the prompt word text, and the SQL statement related to the user is determined based on the prompt word text through the SQL generation model, wherein the target sample corpus represents the sample corpus with the greatest similarity to the user input information in the sample corpus set. In this way, by training a small similarity estimation model, the multi-dimensional information of different questions and database tables is integrated to optimize the similar sample selection process in the context learning prompt word, which can not only adaptively construct prompt words according to user needs, more accurately guide the large model to generate target SQL, but also avoid fine-tuning the parameters of the large language model and reduce the computational cost.

[0162] Those skilled in the art should understand that Figure 5 The implementation functions of each unit in the data processing device shown can be understood by referring to the relevant description of the aforementioned method. Figure 5 The functions of each unit in the data processing device shown may be implemented by a program running on a processor, or may be implemented by a specific logic circuit.

[0163] Figure 6 is a schematic diagram of the structure of a processing device provided in an embodiment of the present application. The processing device may be a terminal device or a network device. Figure 6 The processing device shown includes a processor 601, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0164] Alternatively, if Figure 6 As shown, the processing device may further include a memory 602. The processor 601 may call and run a computer program from the memory 602 to implement the method in the embodiment of the present application.

[0165] The memory 602 may be a separate device independent of the processor 601 , or may be integrated into the processor 601 .

[0166] Alternatively, if Figure 6 As shown, the processing device may further include a transceiver 603, and the processor 601 may control the transceiver 603 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0167] The transceiver 603 may include a transmitter and a receiver. The transceiver 603 may further include an antenna, and the number of antennas may be one or more.

[0168] The processing device may specifically be a cross-domain processing device of an embodiment of the present application, and the processing device may implement the corresponding processes of each method implemented in the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0169] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0170] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0171] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0172] The embodiment of the present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to the processing device in the embodiment of the present application, and the computer program enables the computer to execute the corresponding process of each method implemented in the embodiment of the present application, which will not be described here for the sake of brevity.

[0173] The embodiment of the present application also provides a computer program product, including computer program instructions. The computer program product can be applied to the processing device in the embodiment of the present application, and the computer program instructions enable the computer to execute the corresponding process of each method implemented in the embodiment of the present application, which will not be described here for the sake of brevity.

[0174] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory,) ROM, random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program codes.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a target sample corpus based on the sample corpus set and the user input information through a similarity estimation model; the target sample corpus represents the sample corpus in the sample corpus set that has the greatest similarity to the user input information; Concatenate the target sample corpus with the user input information to obtain a prompt word text; The structured query language SQL generation model is used to determine the user-related SQL statements based on the prompt word text.

2. The method according to claim 1, characterized in that: The method of determining the target sample corpus based on the sample corpus set and user input information by using the similarity estimation model includes: Concatenate each sample corpus in the sample corpus set with the user input information in sequence to obtain multiple text blocks; inputting the plurality of text blocks into the similarity estimation model, and outputting a plurality of similarity ratings through the similarity estimation model; Based on the multiple similarity ratings, the target sample corpus is determined.

3. The method according to claim 2, characterized in that The step of inputting the plurality of text blocks into the similarity estimation model and outputting a plurality of similarity ratings through the similarity estimation model comprises: Inputting the multiple text blocks into the similarity estimation model, and calculating the similarity score between each sample corpus and the user input information through the similarity estimation model to obtain multiple similarity scores; The multiple similarity scores are divided into levels to obtain the multiple similarity ratings, and the multiple similarity ratings are output through the similarity estimation model.

4. The method according to claim 3, characterized in that The similarity rating is negatively correlated with the similarity score; and determining the target sample corpus based on the multiple similarity ratings includes: A minimum similarity rating is selected from the multiple similarity ratings, and a sample corpus in the sample corpus set corresponding to the minimum similarity rating is used as the target sample corpus.

5. The method according to any one of claims 1 to 4, characterized in that The sample corpus set includes a training corpus set and a verification corpus set; the method further includes: Selecting a first sample corpus and a second sample corpus from the training corpus set; Determine a training sample based on the first sample corpus and the second sample corpus; Selecting a third sample corpus from the training corpus set, and selecting a fourth sample corpus from the verification corpus set; Determine a verification sample based on the third sample corpus and the fourth sample corpus; The model is trained using the training samples, and the model is adjusted using the verification samples during the training process to obtain the similarity estimation model.

6. The method according to claim 5, characterized in that The first sample corpus includes a first database structure, a first question, and a first SQL statement, and the second sample corpus includes a second database structure, a second question, and a second SQL statement; The determining of training samples based on the first sample corpus and the second sample corpus includes: The first database structure, the first question and the first SQL statement are concatenated with the second database structure and the second question to obtain a training text block; Calculating a similarity score between the first sample corpus and the second sample corpus; Dividing the similarity scores between the first sample corpus and the second sample corpus according to levels to obtain a similarity rating between the first sample corpus and the second sample corpus; The training sample is determined based on the training text block and the similarity rating between the first sample corpus and the second sample corpus.

7. The method according to claim 6, characterized in that The calculating the similarity score between the first sample corpus and the second sample corpus includes: Determine an SQL keyword set, and based on the SQL keyword set, respectively count the keywords appearing in the first SQL statement and the second SQL statement to obtain a first keyword set and a second keyword set; Counting the number of occurrences of each keyword in the first keyword set in the first SQL statement and the number of occurrences of each keyword in the second keyword set in the second SQL statement to obtain a first number code and a second number code; Based on the first frequency code and the second frequency code, a similarity score between the first sample corpus and the second sample corpus is determined.

8. The method according to claim 7, characterized in that The determining, based on the first frequency code and the second frequency code, a similarity score between the first sample corpus and the second sample corpus includes: The distance between the first order code and the second order code is calculated, and the negative of the distance is used as the similarity score between the first sample corpus and the second sample corpus.

9. The method according to claim 8, characterized in that The method further comprises: Determine the occurrence frequency of corresponding keywords in the SQL keyword set; Determine a weight value of each keyword in the SQL keyword set in distance calculation based on historical knowledge and the occurrence frequency of the keyword; According to the weight value of each keyword in the distance calculation, a weighted distance between the first frequency code and the second frequency code is calculated, and the negative of the weighted distance is used as the similarity score between the first sample corpus and the second sample corpus.

10. The method according to claim 9, characterized in that The prompt word text includes task instructions, target database structure, target question, sample question and corresponding correct SQL statement; the structured query language SQL generation model determines the user-related SQL statement based on the prompt word text, including: Inputting the prompt word text into the SQL generation model, so that the SQL generation model generates a target SQL statement corresponding to the target database structure and the target question according to the task instruction, in combination with the sample question and the correct SQL statement, and outputs the target SQL statement; The target SQL statement is verified and corrected to obtain the user-related SQL statement.

11. A data processing device, characterized in that: The device comprises: An estimation unit, configured to determine a target sample corpus based on a sample corpus set and user input information through a similarity estimation model; the target sample corpus represents a sample corpus in the sample corpus set that has the greatest similarity to the user input information; A concatenation unit, used for concatenating the target sample corpus with the user input information to obtain a prompt word text; A generating unit is used to determine a user-related SQL statement based on the prompt word text by using a structured query language SQL generation model.

12. A processing device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises computer program instructions which cause a computer to execute the method as claimed in any one of claims 1 to 10.

Images